# sushmakl95/spark-performance-optimization-playbook architecture

> Architecture diagram of the GitHub repository sushmakl95/spark-performance-optimization-playbook, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sushmakl95/spark-performance-optimization-playbook
- Repository: https://github.com/sushmakl95/spark-performance-optimization-playbook

## Overview

This repository is a Spark tuning playbook whose main workflow is generating reproducible synthetic data, running before/after benchmarks, and comparing performance. It provides Spark session configurations and benchmark coverage for the documented optimization techniques. The generator-to-session relationship is visible in the sampled source; technique coverage and the command-line workflow are documented in the README. Benchmark implementation details were not sampled, so links from the runner to individual benchmark modules are shown as unverified.

Main entry points: [`benchmarks/run_bench.py`](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/run_bench.py) (Benchmark runner), [`data/generators/ecommerce.py`](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/data/generators/ecommerce.py) (E-commerce generator), [`src/playbook/benchmark.py`](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/src/playbook/benchmark.py) (Benchmark utilities), [`src/playbook/spark_session.py`](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/src/playbook/spark_session.py) (Spark session factories)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Benchmark workflow"]
  node_runner["Benchmark runner<br/>[run_bench.py]"]
  node_harness["Benchmark utilities<br/>[benchmark.py]"]
  node_results["Benchmark comparison<br/>[benchmark.py]"]
end

subgraph group_data["Synthetic data"]
  node_generator["E-commerce generator<br/>[ecommerce.py]"]
  node_parquet[("Generated Parquet data")]
end

subgraph group_runtime["Spark runtime"]
  node_sessions["Spark session factories<br/>[spark_session.py]"]
end

subgraph group_techniques["Optimization benchmarks"]
  node_file_sizing["File sizing"]
  node_partition_pruning["Partition pruning"]
  node_clustering["Z-ORDER and clustering<br/>[bench_zorder.py]"]
  node_broadcast["Broadcast joins"]
  node_aqe["Adaptive execution<br/>[bench_aqe.py]"]
  node_skew["Skew handling"]
  node_executor["Executor tuning"]
  node_caching["Caching and checkpointing<br/>[bench_caching.py]"]
  node_predicate["Predicate pushdown"]
  node_formats["Storage formats"]
  node_liquid["Liquid clustering"]
  node_deletion["Deletion vectors"]
  node_comet["DataFusion Comet<br/>[bench_comet.py]"]
end

node_engineer(("Data engineer"))

node_engineer -->|"runs benchmarks"| node_runner
node_runner -.->|"uses utilities"| node_harness
node_runner -.->|"dispatches"| node_file_sizing
node_runner -.->|"dispatches"| node_partition_pruning
node_runner -.->|"dispatches"| node_clustering
node_runner -.->|"dispatches"| node_broadcast
node_runner -.->|"dispatches"| node_aqe
node_runner -.->|"dispatches"| node_skew
node_runner -.->|"dispatches"| node_executor
node_runner -.->|"dispatches"| node_caching
node_runner -.->|"dispatches"| node_predicate
node_runner -.->|"dispatches"| node_formats
node_runner -.->|"dispatches"| node_liquid
node_runner -.->|"dispatches"| node_deletion
node_runner -.->|"dispatches"| node_comet
node_harness -.->|"produces comparisons"| node_results
node_generator -->|"creates session"| node_sessions
node_generator -->|"writes datasets"| node_parquet

click node_runner "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/run_bench.py"
click node_harness "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/src/playbook/benchmark.py"
click node_results "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/src/playbook/benchmark.py"
click node_generator "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/data/generators/ecommerce.py"
click node_sessions "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/src/playbook/spark_session.py"
click node_file_sizing "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_file_sizing.py"
click node_partition_pruning "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_partition_pruning.py"
click node_clustering "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_zorder.py"
click node_broadcast "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_broadcast_join.py"
click node_aqe "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_aqe.py"
click node_skew "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_skew_handling.py"
click node_executor "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_executor_tuning.py"
click node_caching "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_caching.py"
click node_predicate "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_predicate_pushdown.py"
click node_formats "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_storage_formats.py"
click node_liquid "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_liquid_clustering.py"
click node_deletion "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_deletion_vectors.py"
click node_comet "https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_comet.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_harness,node_results toneBlue
class node_generator,node_parquet toneAmber
class node_sessions toneMint
class node_file_sizing,node_partition_pruning,node_clustering,node_broadcast,node_aqe,node_skew,node_executor,node_caching,node_predicate,node_formats,node_liquid,node_deletion,node_comet toneRose
class node_engineer toneIndigo
```

## Components

### Benchmark workflow

- [Benchmark runner](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/run_bench.py): `benchmarks/run_bench.py`
- [Benchmark utilities](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/src/playbook/benchmark.py): `src/playbook/benchmark.py`
- [Benchmark comparison](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/src/playbook/benchmark.py): `src/playbook/benchmark.py`

### Synthetic data

- [E-commerce generator](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/data/generators/ecommerce.py): `data/generators/ecommerce.py`
- **Generated Parquet data**

### Spark runtime

- [Spark session factories](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/src/playbook/spark_session.py): `src/playbook/spark_session.py`

### Optimization benchmarks

- [File sizing](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_file_sizing.py): `benchmarks/bench_file_sizing.py`
- [Partition pruning](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_partition_pruning.py): `benchmarks/bench_partition_pruning.py`
- [Z-ORDER and clustering](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_zorder.py): `benchmarks/bench_zorder.py`
- [Broadcast joins](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_broadcast_join.py): `benchmarks/bench_broadcast_join.py`
- [Adaptive execution](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_aqe.py): `benchmarks/bench_aqe.py`
- [Skew handling](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_skew_handling.py): `benchmarks/bench_skew_handling.py`
- [Executor tuning](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_executor_tuning.py): `benchmarks/bench_executor_tuning.py`
- [Caching and checkpointing](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_caching.py): `benchmarks/bench_caching.py`
- [Predicate pushdown](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_predicate_pushdown.py): `benchmarks/bench_predicate_pushdown.py`
- [Storage formats](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_storage_formats.py): `benchmarks/bench_storage_formats.py`
- [Liquid clustering](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_liquid_clustering.py): `benchmarks/bench_liquid_clustering.py`
- [Deletion vectors](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_deletion_vectors.py): `benchmarks/bench_deletion_vectors.py`
- [DataFusion Comet](https://github.com/sushmakl95/spark-performance-optimization-playbook/blob/main/benchmarks/bench_comet.py): `benchmarks/bench_comet.py`

### Other components

- **Data engineer**

## Connections

- Data engineer → Benchmark runner: runs benchmarks (evidence: `README.md`)
- Benchmark runner → Benchmark utilities: uses utilities
- Benchmark runner → File sizing: dispatches
- Benchmark runner → Partition pruning: dispatches
- Benchmark runner → Z-ORDER and clustering: dispatches
- Benchmark runner → Broadcast joins: dispatches
- Benchmark runner → Adaptive execution: dispatches
- Benchmark runner → Skew handling: dispatches
- Benchmark runner → Executor tuning: dispatches
- Benchmark runner → Caching and checkpointing: dispatches
- Benchmark runner → Predicate pushdown: dispatches
- Benchmark runner → Storage formats: dispatches
- Benchmark runner → Liquid clustering: dispatches
- Benchmark runner → Deletion vectors: dispatches
- Benchmark runner → DataFusion Comet: dispatches
- Benchmark utilities → Benchmark comparison: produces comparisons
- E-commerce generator → Spark session factories: creates session (evidence: `data/generators/ecommerce.py`)
- E-commerce generator → Generated Parquet data: writes datasets (evidence: `data/generators/ecommerce.py`)