Apache Spark Interview Questions India 2026: RDDs, DataFrames, Optimization, and Data Engineering

Spark core concepts:

1. Spark vs Hadoop MapReduce: Apache MapReduce: each stage writes intermediate results to HDFS (disk). Spark: keeps intermediate results in memory (or spills to disk only when memory is insufficient). For iterative algorithms (ML model training) and multi-stage pipelines, Spark is 10-100x faster than MapReduce. Spark also provides a much richer API (DataFrame, SQL, Streaming, MLlib, GraphX) compared to MapReduce's two-phase (map, reduce) model.

2. Spark architecture: - Driver: the JVM process that runs the main() function; creates the SparkContext/SparkSession; plans the job (builds the DAG); coordinates executors. The driver is the brain. - Cluster Manager: allocates resources across the cluster. Spark supports YARN (Hadoop), Kubernetes, Mesos, and Spark's built-in Standalone cluster manager. - Executors: JVM processes on worker nodes; execute the tasks assigned by the driver; store RDD partitions in memory (cache) and on disk. - Task: the smallest unit of work; executed by one executor on one partition of data.

3. DAG (Directed Acyclic Graph): Spark does not execute transformations immediately. It builds a DAG of transformations and only triggers execution when an action is called. The DAG Scheduler breaks the DAG into stages at shuffle boundaries. Within a stage, tasks on different partitions can run in parallel.

4. Stages and shuffles: A stage boundary occurs at a shuffle. Shuffle: data that must move between executors (e.g., groupBy, join, repartition). Shuffles are expensive (data serialisation, network transfer, deserialisation) and are the primary cause of Spark job slowness. Narrow transformations (filter, map, union): each partition produces one output partition; no shuffle. Wide transformations (groupBy, join, distinct): each output partition may require data from multiple input partitions; triggers a shuffle.

Spark data abstractions:

1. RDD (Resilient Distributed Dataset): The original Spark abstraction. An RDD is a fault-tolerant, distributed collection of objects. Operations: map, filter, reduce, groupBy, join. RDDs are: untyped (Python RDDs have no schema), low-level (no query optimiser), fault-tolerant (RDDs can be recomputed from their lineage if a partition is lost). When to use RDD: fine-grained control over data; when working with unstructured data (logs, raw text); when you need custom partitioning logic. In practice: DataFrames have replaced RDDs for most use cases in Python and Java/Scala.

2. DataFrame: A distributed table with a named schema. Built on top of RDDs but with a query optimiser (Catalyst Optimizer) that rewrites queries for efficiency. Identical API across Python (PySpark), Scala, Java, and R. DataFrames are the primary abstraction for most Spark jobs in 2026.

3. Dataset (Scala/Java only): A typed DataFrame. Provides compile-time type safety that DataFrames lack in Scala/Java. In Python (PySpark), DataFrames are effectively Datasets at runtime (dynamic typing means the distinction matters less).

4. Catalyst Optimizer: The query optimiser that transforms DataFrame operations into an optimised execution plan. Catalyst: analyses the logical plan, applies optimisation rules (predicate pushdown, column pruning, constant folding, join reordering), generates physical plans, and picks the best physical plan. Project Tungsten: Spark's memory management and code generation layer; generates JVM bytecode at runtime for better CPU utilisation.

Optimising Spark jobs:

1. Partitioning: The number of partitions determines the parallelism. Too few: underutilised cluster (some executors idle). Too many: overhead from managing many small tasks. Rule of thumb: 2-4 tasks per CPU core. repartition(n): shuffles data into n partitions (expensive; use when you need to increase partitions or redistribute data). coalesce(n): reduces partitions without a full shuffle (used to reduce output files before writing).

2. Broadcast joins: For a join between a large DataFrame and a small DataFrame: instead of shuffling the large DataFrame, broadcast the small DataFrame to every executor. spark.sql.autoBroadcastJoinThreshold (default 10MB): Spark automatically broadcasts tables smaller than this threshold. Explicitly: dflarge.join(broadcast(dfsmall), on='key').

3. Caching and persistence: df.cache() or df.persist(): stores the DataFrame in memory (or on disk) so subsequent actions on the same DataFrame do not recompute it from scratch. Use when: the same DataFrame is used in multiple downstream operations. Unpersist: df.unpersist() when the cached data is no longer needed; frees memory for other operations.

4. Avoiding data skew: Data skew: some partitions are much larger than others (e.g., a join key that appears disproportionately often, like a default or null value). Causes: straggler tasks (the whole job waits for the slow partition). Solutions: salt the skewed key (add a random suffix to distribute the key across multiple partitions; requires a more complex join logic), increase the number of partitions (reduce the size of each partition), or filter out the skewed key and process it separately.

5. Serialisation: Kryo serialiser is faster and more compact than Java's default serialisation. Configure: spark.serializer=org.apache.spark.serializer.KryoSerializer. For large Spark jobs with complex data structures, Kryo can improve performance significantly.

Spark ecosystem:

1. Spark SQL: Spark SQL allows running SQL queries directly on DataFrames. Create a temp view: df.createOrReplaceTempView('orders'). Query: spark.sql('SELECT customerid, SUM(revenue) FROM orders GROUP BY customerid'). The result is a DataFrame. Full HiveQL and SQL-92 support. Useful for: data analysts who know SQL but not the DataFrame API; for complex queries where SQL is more readable.

2. Structured Streaming: Spark Structured Streaming enables real-time stream processing using the same DataFrame API as batch processing. Read from: Kafka, socket, files. Write to: Kafka, Parquet files, databases. Trigger modes: Trigger.Once() (process all available data then stop; useful for near-real-time batch), Trigger.ProcessingTime('1 minute') (micro-batch every minute), Trigger.Continuous('1 second') (experimental true streaming). Watermarking: handle late-arriving data by defining how long to wait for late data before closing a window.

3. Delta Lake: Delta Lake is an open-source storage layer that adds ACID transactions to Parquet on object storage (S3, ADLS). Key features: ACID transactions (concurrent reads and writes without corruption), schema enforcement (rejects writes that do not match the schema), time travel (query old versions of the table: SELECT * FROM table VERSION AS OF 5), upserts/deletes (MERGE INTO; UPDATE; DELETE). Most Indian data engineering teams using Databricks use Delta Lake as their primary table format.

4. PySpark in data engineering interviews: Common PySpark interview tasks: read a CSV/Parquet file, clean the data (handle nulls, cast types), perform aggregations (groupBy + agg), join two DataFrames, window functions (same as SQL window functions but in DataFrame API), and write output as Parquet. Interviewers at Indian companies (Flipkart, Meesho, Swiggy data teams) often give a take-home or live coding problem that involves reading messy data, cleaning it, and computing a business metric.