Apache Kafka is the backbone of event-driven architectures at Indian fintech, e-commerce, and logistics companies. Swiggy, Razorpay, PhonePe, and Flipkart all rely on Kafka for processing millions of events per second. This guide covers Kafka interview questions for data engineering and backend roles at Indian companies in 2026.
Kafka fundamentals:
1. What is Apache Kafka? Kafka is a distributed event streaming platform. It acts as a durable, high-throughput message bus between producers (services that generate events) and consumers (services that process events). Unlike traditional message queues (RabbitMQ), Kafka persists messages to disk and allows multiple independent consumers to read the same messages at different offsets.
2. Core concepts: - Event (Message): a unit of data with a key, value, timestamp, and optional headers. Events are immutable once written. - Topic: a logical category of events (analogous to a database table or a folder). Topics are durable (events are stored on disk for a configurable retention period, typically 7 days). - Partition: a topic is split into N partitions. Each partition is an ordered, immutable log of events. Partitions enable parallelism (multiple consumers can read from different partitions simultaneously). Events with the same key always go to the same partition (key-based routing ensures ordered processing for a key). - Broker: a Kafka server. A Kafka cluster consists of multiple brokers. Each partition has one leader broker (handles all reads and writes) and zero or more replica brokers (for fault tolerance). - ZooKeeper / KRaft: Kafka traditionally used ZooKeeper for cluster metadata management. KRaft mode (Kafka without ZooKeeper, GA since Kafka 3.3) replaces ZooKeeper with a built-in consensus mechanism.
3. Consumer Groups: A consumer group is a set of consumers that collectively consume a topic. Each partition is consumed by exactly one consumer in the group at any time. This enables parallel processing: a topic with 10 partitions can be consumed by up to 10 consumers in a group concurrently. Multiple consumer groups can independently consume the same topic (each group reads all events independently).
Kafka reliability:
1. Producer delivery guarantees: - At-most-once: the producer sends the message once and does not retry on failure. Fast but messages may be lost. - At-least-once: the producer retries on failure (retries + acks=all). Messages are never lost but may be duplicated if the broker receives the message but the acknowledgment is lost before the producer receives it. - Exactly-once: requires idempotent producers (enable.idempotence=true: each message has a sequence number; the broker deduplicates) + transactional producers (for atomic multi-partition writes).
2. Consumer delivery guarantees: - At-most-once: commit the offset before processing. If the consumer crashes during processing, the event is lost. - At-least-once: process the event, then commit the offset. If the consumer crashes after processing but before committing, the event is processed again. The most common pattern; make your consumers idempotent. - Exactly-once: use Kafka transactions; the consumer processes the event and commits the offset atomically. More complex; used when duplicate processing is unacceptable (financial transactions).
3. Acknowledgments (acks setting): - acks=0: fire and forget; no acknowledgment; fastest; messages may be lost - acks=1: leader acknowledges; fast; messages lost if the leader fails before replication - acks=all (recommended for production): all in-sync replicas acknowledge; most durable; slightly slower
Kafka operations:
1. Partitioning strategy: Key-based partitioning: events with the same key always go to the same partition (kafka ensures ordering within a partition). Use when: order of events for the same entity matters (e.g., all events for the same order_id must be processed in order). Round-robin partitioning: events are distributed evenly across partitions (no key, or null key). Use when: event ordering does not matter; you want maximum parallelism.
2. Consumer lag: Consumer lag = number of unconsumed events in a partition (the difference between the latest offset produced and the committed consumer offset). High consumer lag means the consumers are not keeping up with the producers. Monitor with kafka-consumer-groups.sh --describe or a monitoring tool (Burrow, Confluent Control Center). Fix consumer lag: add more consumers to the group (up to the partition count), optimise consumer processing (make processing faster), or scale horizontally (add more partition replicas).
3. Kafka performance tuning: Producer side: batch.size (larger batches = higher throughput), linger.ms (wait for batch to fill before sending), compression.type (lz4 or snappy: reduces network bandwidth). Consumer side: fetch.min.bytes (wait for at least N bytes before returning; reduces round trips), max.poll.records (maximum events fetched per poll).
4. Replication and fault tolerance: Replication factor: how many copies of each partition exist across brokers. Typically 3 for production. min.insync.replicas: minimum number of replicas that must acknowledge a write (set to 2 with acks=all for durability).
Kafka ecosystem:
1. Kafka Streams: Kafka Streams is a Java library for building stream processing applications. A Kafka Streams application: reads from input topics, transforms/filters/aggregates the data, and writes results to output topics. Runs as a regular JVM process (no separate cluster needed). Key operations: filter, map, flatMap, groupByKey, count, aggregate, join (stream-stream, stream-table). KTable: a changelog stream that represents the current state of a key (the latest value for each key).
2. Kafka Connect: Kafka Connect is a framework for connecting Kafka to external systems. Source Connectors: pull data from external systems (databases, S3, HTTP APIs) into Kafka topics. Sink Connectors: push data from Kafka topics to external systems (Elasticsearch, PostgreSQL, S3). Example: Debezium (a CDC source connector) captures every row-level change in a PostgreSQL database and publishes it to a Kafka topic.
3. Schema Registry: Schema Registry stores and validates Avro (or Protobuf/JSON Schema) schemas for Kafka messages. Ensures producers and consumers agree on the message format. Schema evolution: backward compatible changes (adding optional fields) are allowed; breaking changes (removing fields, changing types) are rejected. Prevents a producer from publishing a message that would break a consumer.
4. Confluent Platform vs Apache Kafka: Apache Kafka is the open-source project. Confluent Platform is a commercial distribution that adds: Confluent Control Center (GUI for monitoring), Schema Registry, Kafka Connect with 200+ connectors, and managed cloud service (Confluent Cloud). At Indian companies: many use Apache Kafka on AWS MSK (Amazon Managed Streaming for Kafka) or on self-managed Kubernetes; Confluent Cloud is used by companies that prefer a fully managed service.