# apache/spark architecture

> Architecture diagram of the GitHub repository apache/spark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/spark
- Repository: https://github.com/apache/spark

## Overview

Apache Spark is a unified analytics engine: users submit workloads through Scala, Java, Python, or R APIs, and Spark executes computation graphs for analytics. Its documented capabilities include SQL/DataFrames, pandas API on Spark, MLlib, GraphX, and Structured Streaming. The graph highlights those product branches and central runtime modules. Relationships among sampled files follow the SOURCE INDEX; broader capability links are documented in the README. Unsampled modules are represented only where the tree supports their paths.

Main entry points: [`core/src/main/scala/org/apache/spark/SparkContext.scala`](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala) (SparkContext), [`core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala`](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala) (Storage RPC endpoint), [`sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala`](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala) (SparkSession), [`core/src/main/scala/org/apache/spark/SparkEnv.scala`](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkEnv.scala) (SparkEnv)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Core runtime"]
  node_sparkcontext["SparkContext<br/>[SparkContext.scala]"]
  node_sparkenv["SparkEnv<br/>[SparkEnv.scala]"]
  node_mapoutput["Map output tracking"]
end

subgraph group_data["Data and shuffle"]
  node_blockmanager["Block manager<br/>[BlockManager.scala]"]
  node_storageendpoint["Storage RPC endpoint"]
  node_storagelevel["Storage levels<br/>[StorageLevel.scala]"]
  node_shufflemanager["Local channel shuffle"]
end

subgraph group_sql["SQL and streaming"]
  node_dataframe["DataFrame API<br/>[dataframe.py]"]
  node_sparksession["SparkSession<br/>[SparkSession.scala]"]
  node_streammanager["Streaming query manager"]
  node_streamapi["Streaming query API"]
  node_connectstream["Connect streaming"]
  node_connecthandler["Connect listener handler"]
end

subgraph group_libraries["Analytics libraries"]
  node_mlpipeline["Scala ML pipeline<br/>[Pipeline.scala]"]
  node_pypipeline["Python ML pipeline<br/>[pipeline.py]"]
  node_graphx["GraphX routing"]
  node_pipelines["Pipeline update context"]
end

subgraph group_support["Shared infrastructure"]
  node_logger["Structured logging<br/>[SparkLogger.java]"]
  node_loggerfactory["Logger factory"]
  node_logging["Scala logging<br/>[Logging.scala]"]
  node_logkeys["Log keys<br/>[LogKeys.java]"]
  node_auth["Transport authentication<br/>[AuthEngine.java]"]
end

node_user(("Workload author"))

node_user -->|"submits workloads"| node_sparkcontext
node_user -->|"uses APIs"| node_dataframe
node_user -->|"builds models"| node_mlpipeline
node_user -->|"builds models"| node_pypipeline
node_user -->|"processes graphs"| node_graphx
node_user -->|"runs streams"| node_streamapi
node_sparkcontext -.->|"initializes runtime"| node_sparkenv
node_sparkcontext -->|"supports SQL workloads"| node_sparksession
node_sparkcontext -->|"supports ML workloads"| node_mlpipeline
node_sparkcontext -->|"supports graph workloads"| node_graphx
node_sparkcontext -->|"creates shuffle writers"| node_shufflemanager
node_storageendpoint -->|"removes blocks"| node_blockmanager
node_storageendpoint -->|"unregisters shuffle"| node_mapoutput
node_storageendpoint -->|"checks managers"| node_sparkenv
node_blockmanager -.->|"uses storage policy"| node_storagelevel
node_sparksession -->|"manages queries"| node_streammanager
node_streammanager -->|"implements API"| node_streamapi
node_connectstream -->|"shares listener API"| node_streamapi
node_connecthandler -->|"handles listener events"| node_connectstream
node_pypipeline -->|"transforms data"| node_dataframe
node_pipelines -->|"uses session"| node_sparksession
node_loggerfactory -->|"creates logger"| node_logger
node_logging -.->|"formats context"| node_logkeys
node_auth -.->|"configures transport"| node_sparkenv

click node_sparkcontext "https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala"
click node_sparkenv "https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkEnv.scala"
click node_mapoutput "https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala"
click node_blockmanager "https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala"
click node_storageendpoint "https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala"
click node_storagelevel "https://github.com/apache/spark/blob/master/common/utils/src/main/scala/org/apache/spark/storage/StorageLevel.scala"
click node_shufflemanager "https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/shuffle/local/pipelined/PipelinedChannelShuffleManager.scala"
click node_dataframe "https://github.com/apache/spark/blob/master/python/pyspark/sql/dataframe.py"
click node_sparksession "https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala"
click node_streammanager "https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/classic/StreamingQueryManager.scala"
click node_streamapi "https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala"
click node_connectstream "https://github.com/apache/spark/blob/master/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/StreamingQueryManager.scala"
click node_connecthandler "https://github.com/apache/spark/blob/master/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectStreamingQueryListenerHandler.scala"
click node_mlpipeline "https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala"
click node_pypipeline "https://github.com/apache/spark/blob/master/python/pyspark/ml/pipeline.py"
click node_graphx "https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/impl/RoutingTablePartition.scala"
click node_pipelines "https://github.com/apache/spark/blob/master/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/PipelineUpdateContextImpl.scala"
click node_logger "https://github.com/apache/spark/blob/master/common/utils-java/src/main/java/org/apache/spark/internal/SparkLogger.java"
click node_loggerfactory "https://github.com/apache/spark/blob/master/common/utils-java/src/main/java/org/apache/spark/internal/SparkLoggerFactory.java"
click node_logging "https://github.com/apache/spark/blob/master/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala"
click node_logkeys "https://github.com/apache/spark/blob/master/common/utils-java/src/main/java/org/apache/spark/internal/LogKeys.java"
click node_auth "https://github.com/apache/spark/blob/master/common/network-common/src/main/java/org/apache/spark/network/crypto/AuthEngine.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sparkcontext,node_sparkenv,node_mapoutput toneBlue
class node_blockmanager,node_storageendpoint,node_storagelevel,node_shufflemanager toneAmber
class node_dataframe,node_sparksession,node_streammanager,node_streamapi,node_connectstream,node_connecthandler toneMint
class node_mlpipeline,node_pypipeline,node_graphx,node_pipelines toneRose
class node_logger,node_loggerfactory,node_logging,node_logkeys,node_auth,node_user toneIndigo
```

## Components

### Core runtime

- [SparkContext](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala): `core/src/main/scala/org/apache/spark/SparkContext.scala`
- [SparkEnv](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkEnv.scala): `core/src/main/scala/org/apache/spark/SparkEnv.scala`
- [Map output tracking](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala): `core/src/main/scala/org/apache/spark/MapOutputTracker.scala`

### Data and shuffle

- [Block manager](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala): `core/src/main/scala/org/apache/spark/storage/BlockManager.scala`
- [Storage RPC endpoint](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala): `core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala`
- [Storage levels](https://github.com/apache/spark/blob/master/common/utils/src/main/scala/org/apache/spark/storage/StorageLevel.scala): `common/utils/src/main/scala/org/apache/spark/storage/StorageLevel.scala`
- [Local channel shuffle](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/shuffle/local/pipelined/PipelinedChannelShuffleManager.scala): `core/src/main/scala/org/apache/spark/shuffle/local/pipelined/PipelinedChannelShuffleManager.scala`

### SQL and streaming

- [DataFrame API](https://github.com/apache/spark/blob/master/python/pyspark/sql/dataframe.py): `python/pyspark/sql/dataframe.py`
- [SparkSession](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala): `sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala`
- [Streaming query manager](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/classic/StreamingQueryManager.scala): `sql/core/src/main/scala/org/apache/spark/sql/classic/StreamingQueryManager.scala`
- [Streaming query API](https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala): `sql/api/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala`
- [Connect streaming](https://github.com/apache/spark/blob/master/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/StreamingQueryManager.scala): `sql/connect/common/src/main/scala/org/apache/spark/sql/connect/StreamingQueryManager.scala`
- [Connect listener handler](https://github.com/apache/spark/blob/master/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectStreamingQueryListenerHandler.scala): `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectStreamingQueryListenerHandler.scala`

### Analytics libraries

- [Scala ML pipeline](https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala): `mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala`
- [Python ML pipeline](https://github.com/apache/spark/blob/master/python/pyspark/ml/pipeline.py): `python/pyspark/ml/pipeline.py`
- [GraphX routing](https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/impl/RoutingTablePartition.scala): `graphx/src/main/scala/org/apache/spark/graphx/impl/RoutingTablePartition.scala`
- [Pipeline update context](https://github.com/apache/spark/blob/master/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/PipelineUpdateContextImpl.scala): `sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/PipelineUpdateContextImpl.scala`

### Shared infrastructure

- [Structured logging](https://github.com/apache/spark/blob/master/common/utils-java/src/main/java/org/apache/spark/internal/SparkLogger.java): `common/utils-java/src/main/java/org/apache/spark/internal/SparkLogger.java`
- [Logger factory](https://github.com/apache/spark/blob/master/common/utils-java/src/main/java/org/apache/spark/internal/SparkLoggerFactory.java): `common/utils-java/src/main/java/org/apache/spark/internal/SparkLoggerFactory.java`
- [Scala logging](https://github.com/apache/spark/blob/master/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala): `common/utils/src/main/scala/org/apache/spark/internal/Logging.scala`
- [Log keys](https://github.com/apache/spark/blob/master/common/utils-java/src/main/java/org/apache/spark/internal/LogKeys.java): `common/utils-java/src/main/java/org/apache/spark/internal/LogKeys.java`
- [Transport authentication](https://github.com/apache/spark/blob/master/common/network-common/src/main/java/org/apache/spark/network/crypto/AuthEngine.java): `common/network-common/src/main/java/org/apache/spark/network/crypto/AuthEngine.java`

### Other components

- **Workload author**

## Connections

- Workload author → SparkContext: submits workloads
- Workload author → DataFrame API: uses APIs
- Workload author → Scala ML pipeline: builds models
- Workload author → Python ML pipeline: builds models
- Workload author → GraphX routing: processes graphs
- Workload author → Streaming query API: runs streams
- SparkContext → SparkEnv: initializes runtime (evidence: `core/src/main/scala/org/apache/spark/SparkEnv.scala`)
- SparkContext → SparkSession: supports SQL workloads
- SparkContext → Scala ML pipeline: supports ML workloads (evidence: `mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala`)
- SparkContext → GraphX routing: supports graph workloads
- SparkContext → Local channel shuffle: creates shuffle writers (evidence: `core/src/main/scala/org/apache/spark/shuffle/local/pipelined/PipelinedChannelShuffleManager.scala`)
- Storage RPC endpoint → Block manager: removes blocks (evidence: `core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala`)
- Storage RPC endpoint → Map output tracking: unregisters shuffle (evidence: `core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala`)
- Storage RPC endpoint → SparkEnv: checks managers (evidence: `core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala`)
- Block manager → Storage levels: uses storage policy
- SparkSession → Streaming query manager: manages queries (evidence: `sql/core/src/main/scala/org/apache/spark/sql/classic/StreamingQueryManager.scala`)
- Streaming query manager → Streaming query API: implements API (evidence: `sql/core/src/main/scala/org/apache/spark/sql/classic/StreamingQueryManager.scala`)
- Connect streaming → Streaming query API: shares listener API (evidence: `sql/connect/common/src/main/scala/org/apache/spark/sql/connect/StreamingQueryManager.scala`)
- Connect listener handler → Connect streaming: handles listener events (evidence: `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectStreamingQueryListenerHandler.scala`)
- Python ML pipeline → DataFrame API: transforms data (evidence: `python/pyspark/ml/pipeline.py`)
- Pipeline update context → SparkSession: uses session (evidence: `sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/PipelineUpdateContextImpl.scala`)
- Logger factory → Structured logging: creates logger (evidence: `common/utils-java/src/main/java/org/apache/spark/internal/SparkLoggerFactory.java`)
- Scala logging → Log keys: formats context (evidence: `common/utils/src/main/scala/org/apache/spark/internal/Logging.scala`)
- Transport authentication → SparkEnv: configures transport