Scala is the language of choice for data engineering roles at Walmart Global Tech India, Target India, and Flipkart, where Apache Spark is used extensively for large-scale data pipelines. If you are targeting a Scala or Spark position in India, you need to demonstrate fluency in functional programming, Scala's type system, and distributed data processing concepts.
Scala vs Java and Functional Programming Fundamentals
Scala runs on the JVM and interoperates fully with Java, but it adds powerful functional programming features that change how you write code.
Scala vs Java:
- val vs var: val declares an immutable reference (like Java final); var declares a mutable reference. Prefer val — immutability prevents bugs in concurrent code
- Type inference: Scala infers types so you do not need to write them explicitly: val name = 'Alice' (type String is inferred)
- Everything is an expression: if/else, match, and try/catch all return values in Scala
- No checked exceptions: Scala does not have checked exceptions (unlike Java)
- Semicolons are optional in Scala
Functional programming concepts:
- Pure functions: a function with no side effects; given the same inputs, always returns the same output. Easier to test and reason about
- Immutability: prefer immutable data structures; Scala collections are immutable by default (List, Map, Set from scala.collection.immutable)
- Higher-order functions: functions that take other functions as parameters or return functions: list.map(x => x * 2), list.filter(x => x > 0), list.flatMap, list.fold
- Function composition: combine small functions into larger ones: val transform = f andThen g (apply f first, then g)
- Currying: transforming a function with multiple arguments into a sequence of functions each taking one argument
Pattern matching: 11. match expressions are Scala's powerful switch statement that can deconstruct data structures: case class Point(x: Int, y: Int) point match { case Point(0, 0) => 'origin' case Point(x, 0) => s'on x-axis at $x' case Point(0, y) => s'on y-axis at $y' case Point(x, y) => s'at ($x, $y)' } 12. Pattern matching works on lists, tuples, case classes, and sealed trait hierarchies 13. Sealed traits: a sealed trait can only be extended in the same file; the compiler warns if a match is not exhaustive
Case Classes, Traits, and the Scala Type System
Case classes and traits are the building blocks of Scala data modelling and polymorphism.
Case classes:
- case class Person(name: String, age: Int) — immutable data holder
- Automatically generated: equals (structural equality), hashCode, toString, copy, and companion object with apply/unapply
- copy method: val older = person.copy(age = person.age + 1) — creates a new instance with one field changed
- No need for new keyword: val p = Person('Alice', 30)
- Case classes work perfectly with pattern matching because of the unapply extractor
Traits:
- Traits are like Java interfaces but can have concrete method implementations and state
- A class can mix in multiple traits (unlike Java's single-class inheritance): class Dog extends Animal with Runnable with Serializable
- Trait linearisation: Scala resolves multiple trait inheritance via a deterministic linearisation order (right to left in the extends clause)
- Abstract methods in traits must be implemented by concrete classes or other traits
Option, Either, and Try for error handling:
- Option[A]: represents an optional value; Some(value) or None; avoids null pointer exceptions
- map, flatMap, and getOrElse work on Option: option.getOrElse(defaultValue)
- Either[L, R]: Left conventionally represents an error; Right represents success; used for functional error handling
- Try[A]: wraps a computation that may throw an exception; Success(value) or Failure(exception)
Collections:
- Immutable collections (default): List (linked list; fast prepend), Vector (indexed; fast random access), Map, Set
- Mutable collections (scala.collection.mutable): ArrayBuffer, HashMap — use sparingly
- Common transformations: map (transform each element), filter (keep elements matching a predicate), flatMap (map then flatten), fold/reduce (aggregate), groupBy (group elements by a key), zip (combine two collections element-by-element)
Akka Basics and Apache Spark with Scala
Akka and Spark are the two most common Scala frameworks tested in Indian data engineering and backend interviews.
Akka Actor Model basics:
- The Actor Model is a concurrency model where actors are the basic unit of computation; actors communicate only by sending messages (no shared mutable state)
- ActorSystem: the container for all actors in an application
- ActorRef: a reference to an actor; you send messages to an ActorRef, not directly to an actor
- tell (!): fire-and-forget message sending: actorRef ! message
- ask (?): sends a message and returns a Future with the response
- Actors process messages from their mailbox sequentially (one at a time); no synchronisation is needed inside an actor
- Supervision: parent actors supervise child actors; a supervisor decides what to do when a child fails (restart, stop, escalate)
- Akka Streams: reactive stream processing on top of the actor model; provides backpressure
Apache Spark with Scala:
- SparkSession: the entry point for all Spark functionality; replaces SparkContext + SQLContext + HiveContext
- RDDs (Resilient Distributed Datasets): the low-level distributed collection API; fault-tolerant; partitioned across the cluster
- DataFrames: distributed collections of rows with named columns; similar to a relational table; optimised by the Catalyst query optimiser
- Datasets[T]: type-safe DataFrames specific to Scala and Java; Dataset[Row] is the same as DataFrame
- Transformations are lazy: map, filter, groupBy build a logical plan; no computation happens until an action is called
- Actions trigger execution: collect, count, show, write
- Narrow vs wide transformations: narrow (map, filter) do not require data shuffle; wide (groupBy, join, repartition) require a shuffle across the network — the most expensive Spark operation
- Partitioning: repartition(n) creates exactly n partitions (involves a full shuffle); coalesce(n) reduces partitions without a full shuffle (use to reduce partition count before writing)
- Caching: df.cache() or df.persist(StorageLevel.MEMORYANDDISK) stores a DataFrame in memory for reuse across multiple actions
- Broadcast variables: broadcastVar = sc.broadcast(largeMap) — sends a read-only variable to all executor nodes once, avoiding re-sending it with every task
Frequently asked questions
Explore more