How sushmakl95/spark-performance-optimization-playbook works
This repository is a Spark tuning playbook whose main workflow is generating reproducible synthetic data, running before/after benchmarks, and comparing performance. It provides Spark session configurations and benchmark coverage for the documented optimization techniques. The generator-to-session relationship is visible in the sampled source; technique coverage and the command-line workflow are documented in the README. Benchmark implementation details were not sampled, so links from the runner to individual benchmark modules are shown as unverified.
Main entry points: run_bench.py Benchmark runner · ecommerce.py E-commerce generator · benchmark.py Benchmark utilities · spark_session.py Spark session factories
Components 20
Benchmark workflow
- Benchmark runner
benchmarks/run_bench.py - Benchmark utilities
src/playbook/benchmark.py - Benchmark comparison
src/playbook/benchmark.py
Synthetic data
- E-commerce generator
data/generators/ecommerce.py - Generated Parquet data
Spark runtime
- Spark session factories
src/playbook/spark_session.py
Optimization benchmarks
- File sizing
benchmarks/bench_file_sizing.py - Partition pruning
benchmarks/bench_partition_pruning.py - Z-ORDER and clustering
benchmarks/bench_zorder.py - Broadcast joins
benchmarks/bench_broadcast_join.py - Adaptive execution
benchmarks/bench_aqe.py - Skew handling
benchmarks/bench_skew_handling.py - Executor tuning
benchmarks/bench_executor_tuning.py - Caching and checkpointing
benchmarks/bench_caching.py - Predicate pushdown
benchmarks/bench_predicate_pushdown.py - Storage formats
benchmarks/bench_storage_formats.py - Liquid clustering
benchmarks/bench_liquid_clustering.py - Deletion vectors
benchmarks/bench_deletion_vectors.py - DataFusion Comet
benchmarks/bench_comet.py
Other components
- Data engineer
Connections 18
- Data engineer → Benchmark runner: runs benchmarks
README.md - Benchmark runner → Benchmark utilities: uses utilities
- Benchmark runner → File sizing: dispatches
- Benchmark runner → Partition pruning: dispatches
- Benchmark runner → Z-ORDER and clustering: dispatches
- Benchmark runner → Broadcast joins: dispatches
- Benchmark runner → Adaptive execution: dispatches
- Benchmark runner → Skew handling: dispatches
- Benchmark runner → Executor tuning: dispatches
- Benchmark runner → Caching and checkpointing: dispatches
- Benchmark runner → Predicate pushdown: dispatches
- Benchmark runner → Storage formats: dispatches
- Benchmark runner → Liquid clustering: dispatches
- Benchmark runner → Deletion vectors: dispatches
- Benchmark runner → DataFusion Comet: dispatches
- Benchmark utilities → Benchmark comparison: produces comparisons
- E-commerce generator → Spark session factories: creates session
data/generators/ecommerce.py - E-commerce generator → Generated Parquet data: writes datasets
data/generators/ecommerce.py