Data Engineer Interview Questions India 2026: Spark, Kafka, Airflow, Data Pipelines, and SQL at Scale

Apache Spark interview questions:

1. 'What is the difference between a transformation and an action in Spark?' Transformation: creates a new RDD/DataFrame from an existing one; lazy (not executed immediately). Examples: map, filter, groupBy, join, select, withColumn. Action: triggers execution of all pending lazy transformations; returns a result to the driver or writes to storage. Examples: count(), collect(), show(), write.save().

2. 'What is a Spark DAG?' When you call an action, Spark builds a DAG (Directed Acyclic Graph) of transformations. The DAG is divided into stages at shuffle boundaries (groupBy, join). Tasks within a stage run in parallel across the cluster.

3. 'What is the difference between repartition() and coalesce()?' repartition(N): creates exactly N partitions; full shuffle; use to increase or decrease partitions. coalesce(N): reduces partitions without a full shuffle; more efficient when reducing only.

4. 'What is a broadcast join?' The smaller table is broadcast to all executors so the join happens locally without a shuffle. Use when one table is small enough to fit in memory (under spark.sql.autoBroadcastJoinThreshold, default 10MB).

5. 'What is data skew and how do you fix it?' Data skew: some partitions have much more data than others (e.g., one user_id has 80% of rows), leading to straggler tasks. Solutions: salting (add a random prefix to the join key to distribute skewed data), AQE (Adaptive Query Execution in Spark 3+, which automatically splits large partitions and applies broadcast joins where possible).

6. 'What is Spark caching?' df.cache() stores the DataFrame in memory across transformations so it is not recomputed when accessed multiple times. Use when the same DataFrame is reused in multiple downstream transformations or actions.

Kafka interview questions:

1. 'What is Apache Kafka?' A distributed event streaming platform acting as a durable, high-throughput message queue between producers (write events) and consumers (process events). Use cases: decoupling microservices, real-time analytics pipelines, event sourcing and audit logs.

2. 'What is a Kafka consumer group?' A set of consumers that jointly consume a topic. Each partition is consumed by exactly one consumer in the group at a time (parallelism = number of partitions). Multiple consumer groups independently consume the same topic.

3. 'What is Kafka offset?' A unique, monotonically increasing integer that identifies each message within a partition. Consumers track their offset so they know which messages they have already processed.

4. 'What is the difference between at-most-once, at-least-once, and exactly-once delivery in Kafka?' At-most-once: offsets committed before processing; if the consumer crashes after committing but before processing, the message is lost. At-least-once: offsets committed after processing; if the consumer crashes after processing but before committing, the message is reprocessed. Exactly-once: Kafka Transactions API guarantees each message is processed exactly once.

Airflow interview questions:

5. 'What is Apache Airflow?' A workflow orchestration platform. Pipelines are defined as DAGs in Python; each node is a task (Python operator, BashOperator, SparkSubmitOperator). Airflow handles scheduling (cron-based), dependency management, retries, and a web UI for monitoring. Used at most Indian data teams for ETL/ELT pipeline orchestration.