How apache/spark works
Apache Spark is a unified analytics engine: users submit workloads through Scala, Java, Python, or R APIs, and Spark executes computation graphs for analytics. Its documented capabilities include SQL/DataFrames, pandas API on Spark, MLlib, GraphX, and Structured Streaming. The graph highlights those product branches and central runtime modules. Relationships among sampled files follow the SOURCE INDEX; broader capability links are documented in the README. Unsampled modules are represented only where the tree supports their paths.
Main entry points: SparkContext.scala SparkContext · BlockManagerStorageEndpoint.scala Storage RPC endpoint · SparkSession.scala SparkSession · SparkEnv.scala SparkEnv
Components 23
Core runtime
- SparkContext
core/src/main/scala/org/apache/spark/SparkContext.scala - SparkEnv
core/src/main/scala/org/apache/spark/SparkEnv.scala - Map output tracking
core/src/main/scala/org/apache/spark/MapOutputTracker.scala
Data and shuffle
- Block manager
core/src/main/scala/org/apache/spark/storage/BlockManager.scala - Storage RPC endpoint
core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala - Storage levels
common/utils/src/main/scala/org/apache/spark/storage/StorageLevel.scala - Local channel shuffle
core/src/main/scala/org/apache/spark/shuffle/local/pipelined/PipelinedChannelShuffleManager.scala
SQL and streaming
- DataFrame API
python/pyspark/sql/dataframe.py - SparkSession
sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala - Streaming query manager
sql/core/src/main/scala/org/apache/spark/sql/classic/StreamingQueryManager.scala - Streaming query API
sql/api/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala - Connect streaming
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/StreamingQueryManager.scala - Connect listener handler
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectStreamingQueryListenerHandler.scala
Analytics libraries
- Scala ML pipeline
mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala - Python ML pipeline
python/pyspark/ml/pipeline.py - GraphX routing
graphx/src/main/scala/org/apache/spark/graphx/impl/RoutingTablePartition.scala - Pipeline update context
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/PipelineUpdateContextImpl.scala
Shared infrastructure
- Structured logging
common/utils-java/src/main/java/org/apache/spark/internal/SparkLogger.java - Logger factory
common/utils-java/src/main/java/org/apache/spark/internal/SparkLoggerFactory.java - Scala logging
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala - Log keys
common/utils-java/src/main/java/org/apache/spark/internal/LogKeys.java - Transport authentication
common/network-common/src/main/java/org/apache/spark/network/crypto/AuthEngine.java
Other components
- Workload author
Connections 24
- Workload author → SparkContext: submits workloads
- Workload author → DataFrame API: uses APIs
- Workload author → Scala ML pipeline: builds models
- Workload author → Python ML pipeline: builds models
- Workload author → GraphX routing: processes graphs
- Workload author → Streaming query API: runs streams
- SparkContext → SparkEnv: initializes runtime
core/src/main/scala/org/apache/spark/SparkEnv.scala - SparkContext → SparkSession: supports SQL workloads
- SparkContext → Scala ML pipeline: supports ML workloads
mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala - SparkContext → GraphX routing: supports graph workloads
- SparkContext → Local channel shuffle: creates shuffle writers
core/src/main/scala/org/apache/spark/shuffle/local/pipelined/PipelinedChannelShuffleManager.scala - Storage RPC endpoint → Block manager: removes blocks
core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala - Storage RPC endpoint → Map output tracking: unregisters shuffle
core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala - Storage RPC endpoint → SparkEnv: checks managers
core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala - Block manager → Storage levels: uses storage policy
- SparkSession → Streaming query manager: manages queries
sql/core/src/main/scala/org/apache/spark/sql/classic/StreamingQueryManager.scala - Streaming query manager → Streaming query API: implements API
sql/core/src/main/scala/org/apache/spark/sql/classic/StreamingQueryManager.scala - Connect streaming → Streaming query API: shares listener API
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/StreamingQueryManager.scala - Connect listener handler → Connect streaming: handles listener events
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectStreamingQueryListenerHandler.scala - Python ML pipeline → DataFrame API: transforms data
python/pyspark/ml/pipeline.py - Pipeline update context → SparkSession: uses session
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/PipelineUpdateContextImpl.scala - Logger factory → Structured logging: creates logger
common/utils-java/src/main/java/org/apache/spark/internal/SparkLoggerFactory.java - Scala logging → Log keys: formats context
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala - Transport authentication → SparkEnv: configures transport