Apache Spark internals: Reading notes

These are my reading notes from Learning Spark, 2nd Edition (Damji et al., O’Reilly). They cover the unified analytics engine’s architecture, the transition from RDDs to Structured APIs, and the mechanics of the Catalyst optimizer.

2026 Perspective: While the 2nd Edition is the foundational text, the Spark ecosystem has since standardized on Spark Connect. The tight coupling between the client and the JVM is gone; in 2026, we mostly interact with Spark via thin gRPC clients, enabling better IDE integration and remote development without local Spark setups.

Core Architecture

Spark 2.x and 3.x moved away from the opaque RDD-based execution toward a structured engine. The core components include:

• Modularity: Support for Spark SQL, MLlib (pipelines), Structured Streaming, and GraphX.
• Unified Analytics: A stream is viewed as a continuously growing table; the same SQL/DataFrame API applies to both batch and streaming.
• Distributed Execution:
  • Driver: Orchestrates parallel execution, creates the SparkSession, and converts the application into a Directed Acyclic Graph (DAG).
  • Executors: Run tasks on the cluster nodes.
  • Cluster Manager: YARN, Kubernetes, Mesos, or Standalone mode.

Sidenote: In 2026, Kubernetes has effectively won the Cluster Manager war for on-prem and cloud-native deployments, with YARN largely relegated to legacy Hadoop clusters.

Spark Application Concepts

A Spark application is broken down into a hierarchy of execution:

1. Job: A parallel computation triggered by an Action (e.g., count(), save()).
2. Stage: A job is divided into stages based on Shuffle boundaries (Wide dependencies). Narrow dependencies (like map or filter) can stay within a single stage.
3. Task: The smallest unit of work, sent to an executor. One task maps to one core and one partition.

Transformations vs. Actions

Spark is lazy. Transformations (like select, join, filter) build up a logical plan (lineage) but don’t execute until an Action (like show, collect) is called. This allows the engine to optimize the entire DAG before touching data.

Structured APIs: DataFrames and Datasets

The RDD API is “dumb” because Spark cannot see the intent of your code; it just sees an opaque lambda function. The Structured APIs (DataFrame and Dataset) provide a schema and typed operations that Spark can optimize.

API	Type	Language Support
RDD	Low-level / Unstructured	Scala, Java, Python, R
DataFrame	Untyped (Dataset[Row])	Scala, Java, Python, R
Dataset	Strongly typed (Dataset[T])	Scala, Java (JVM only)

2026 Perspective: Python has become the dominant language for Spark. With vectorized UDFs and the maturity of the Pandas API on Spark, the “performance penalty” of Python is essentially a thing of the past for most data engineering workloads.

Schema-on-read

While Spark can infer schemas from CSV/JSON, it is significantly faster and safer to define them upfront using a StructType or a DDL string. This avoids an extra scan of the data during the planning phase.

Catalyst and Tungsten

This is the “magic” that makes Structured APIs faster than RDDs.

1. Catalyst Optimizer: Converts a logical plan into an optimized physical plan. It performs:
   • Predicate Pushdown: Moving filters as close to the data source as possible.
   • Column Pruning: Only reading the columns needed for the query.
   • Constant Folding: Pre-calculating expressions.
2. Tungsten Engine: A physical execution layer that focuses on memory management and code generation. It avoids the overhead of the JVM object model and garbage collection by using an “off-heap” binary format.

Spark SQL and Data Sources

Spark supports a variety of formats through the DataFrameReader and DataFrameWriter.

• Parquet: The gold standard for Spark. It is columnar, stores schemas/metadata, and supports predicate pushdown.
• ORC: Similar to Parquet, often used in Hive environments. Supports a vectorized reader for faster scans.
• Avro: Row-based format, excellent for write-heavy workloads or message buses like Kafka.
• Binary/Image: Support for reading raw files into a schema containing the path and binary content.

Sidenote: By 2026, Open Table Formats (Delta Lake, Apache Iceberg, and Hudi) have largely replaced “raw” Parquet as the default storage layer. We no longer think in files; we think in ACID-compliant tables with versioning and time-travel.

Managed vs. Unmanaged Tables

• Managed: Spark (via Hive Metastore) manages both the metadata and the data. Dropping the table deletes the files.
• Unmanaged (EXTERNAL): Spark only manages the metadata. Dropping the table leaves the files intact on HDFS/S3.

Performance Tuning

Maximizing Parallelism

Partitions are the atomic units of parallelism. If you have 100 cores but only 8 partitions, 92 cores stay idle.

• spark.sql.shuffle.partitions: Defaults to 200. For smaller workloads, this is often too high (generating many small files). For massive joins, it might be too low.
• repartition() vs coalesce(): Repartition triggers a full shuffle to increase/decrease partitions. Coalesce only decreases partitions and tries to avoid a full shuffle.

2026 Perspective: Adaptive Query Execution (AQE) is now mature and on by default. In most cases, you no longer manually tune spark.sql.shuffle.partitions; Spark merges small partitions or splits large ones dynamically at runtime based on actual data statistics.

Joins

1. Broadcast Hash Join (BHJ): If one side is small (default <10MB), Spark broadcasts it to all executors. No shuffle, very fast.
2. Shuffle Sort Merge Join (SMJ): The default for joining two large tables. Both sides are shuffled, sorted by the join key, and merged.

Caching

• cache(): An alias for persist(StorageLevel.MEMORY_AND_DISK).
• Note: Caching is also lazy. The data isn’t cached until the first action is executed. Use it when you plan to access the same DataFrame multiple times in iterative loops or different branches of a pipeline.

Structured Streaming

Structured Streaming treats a live stream as an unbounded table.

• Checkpointing: Mandatory for failure recovery. It stores the offsets in HDFS-compatible storage.
• Output Modes:
  • Append: Only new rows are written.
  • Update: Only changed rows are updated (useful for sinks like RDBMS).
  • Complete: The entire table is rewritten (useful for aggregations).
• Watermarking: Defines how long the engine should wait for “late” data before discarding the state for a specific time window.

---

These notes are based on Learning Spark by Jules S. Damji, Brooke Wenig, Tathagata Das, and Denny Lee.