Data Engineer Interview Questions India 2026: Spark, Kafka, dbt and Pipeline Design

Spark is the dominant distributed processing framework for batch data pipelines in India. Architecture: the Spark application has a Driver (the main program, runs on one machine, coordinates the job) and Executors (worker processes distributed across the cluster, do the actual computation). The Cluster Manager (YARN, Kubernetes, or Spark Standalone) allocates resources. DAG (Directed Acyclic Graph): Spark builds a DAG of operations before executing. This allows the Catalyst optimiser to combine operations, push filters early (predicate pushdown), and eliminate unnecessary shuffles. RDD vs DataFrame vs Dataset: RDD (Resilient Distributed Dataset) is the low-level abstraction. DataFrames have a schema and are optimised by Catalyst. Datasets are typed DataFrames (Scala/Java). Use DataFrames in Python (PySpark): they give you SQL-like optimisations. Transformations vs actions: transformations (map, filter, select, groupBy, join) are lazy: they build the DAG but do not execute. Actions (count, show, collect, write) trigger execution. Wide vs narrow transformations: narrow transformations (map, filter) can execute within a partition without data movement. Wide transformations (groupBy, join, distinct) require a shuffle: data is redistributed across the network. Shuffles are the primary performance bottleneck in Spark jobs. Optimisation techniques: broadcast join for small tables (avoids a shuffle by sending the small table to each executor). Partition tuning: too few partitions underutilises the cluster; too many creates overhead. Skew handling: when some partitions have much more data than others (a single key like 'category = electronics' having 80% of records), performance degrades. Solutions: salting (add a random prefix to the key, join both sides, then aggregate).

Kafka is the standard event streaming platform at Indian product companies for real-time data pipelines. Architecture: Brokers (servers that store and serve messages), Topics (named log of messages), Partitions (topics are split into partitions for parallelism, each partition is ordered), Consumer Groups (each consumer group reads every message once across its members). Producer sends to a partition using a partition key (default: round-robin if no key). Message delivery semantics: At-most-once (no retries, messages may be lost). At-least-once (retries on failure, messages may be delivered twice, requires idempotent consumers). Exactly-once (Kafka transactions or Kafka Streams with enable.idempotence=true). Use exactly-once for financial data; at-least-once is sufficient for analytics events. Consumer lag: the difference between the latest offset and the committed offset. High lag means consumers are processing too slowly. Fix: add more consumers (up to the partition count), optimise consumer code, or reduce message production rate. Replication: each partition has a leader and N replicas. Replication factor of 3 is standard for production. A broker failure causes the controller to elect a new leader from the in-sync replicas (ISR). Kafka Connect: for ingesting data from databases into Kafka (Debezium for Change Data Capture: streams every INSERT, UPDATE, DELETE from Postgres or MySQL into a Kafka topic). Schema Registry: prevents consumer-producer contract breaks by enforcing that all messages on a topic conform to a registered schema (Avro or Protobuf). Common interview question: 'You have a Kafka topic with 10 partitions. You start a consumer group with 15 consumers. What happens?' Answer: 5 consumers remain idle, since the maximum parallelism equals the partition count.

Data warehouse design and transformation tooling are the third pillar of data engineering interviews in India. dbt (data build tool): transforms raw data in your data warehouse using SQL models. dbt models are SELECT statements that dbt materialises as tables or views. dbt tests: schema tests (unique, notnull, acceptedvalues, relationships) that run automatically after each transformation. dbt lineage: dbt builds a DAG of model dependencies automatically using ref() references. dbt documentation: auto-generates a data dictionary from model descriptions. Why dbt: enables version control for SQL transformations (models in Git), automated testing, and lineage documentation for audit. Data warehouse design: dimensional modelling (Kimball methodology) is still widely used in India. Fact tables store events (orders, transactions, page views). Dimension tables store attributes (customers, products, geography). Star schema: one fact table surrounded by dimension tables, optimised for analytical queries. Snowflake schema: dimensions are further normalised. Star schema is preferred for performance; snowflake for storage. Common warehouses in India: Google BigQuery (Swiggy, Zomato, Flipkart data teams), Snowflake (growing adoption), Redshift (AWS-native companies), ClickHouse (for high-throughput real-time analytics). Pipeline orchestration: Airflow (most commonly used in India), Prefect, Dagster. DAG definition: a Python file that defines tasks and dependencies. Common Airflow patterns: sensor tasks (wait for a file or API condition before proceeding), backfill (rerun historical data for a date range), XCom for passing small data between tasks.

Data engineering is one of India's highest-growth roles in 2026 with compensation catching up to backend software engineering. 2026 salary benchmarks: Junior data engineer (0-2 years): Rs 10-25 LPA at product companies. Mid-level (2-5 years): Rs 25-60 LPA. Senior data engineer (5-8 years): Rs 55-105 LPA. Staff/Principal data engineer: Rs 95-170 LPA at FAANG India. Company-specific demand: every company with significant data volume (Flipkart, Swiggy, PhonePe, Meesho, Zepto, MakeMyTrip, Myntra) has data engineering teams of 20-100+ engineers. The roles vary: some companies focus on batch processing (Spark on EMR for nightly reports), others on real-time streaming (Flink or Kafka Streams for sub-second analytics). Skills that command a premium: ClickHouse experience (high-demand, lower supply, used for real-time dashboards at Swiggy, Zepto, Blinkit). Apache Flink (real-time stream processing, more complex than Kafka Streams). Data mesh knowledge (federated data ownership model adopted by several large Indian companies). Career paths from data engineering: Staff Data Engineer (technical depth, designing data platforms), Engineering Manager for Data Platform, Analytics Engineer (SQL-centric track, dbt specialisation), Data Architect. The data-to-ML bridge: data engineers who also understand MLOps (feature stores, model serving infrastructure) are in extremely high demand at companies with active ML programmes (Flipkart, Myntra, Swiggy). This hybrid role commands a 20-30% premium.