How amtorigami/indrive_spark works
This repository is a Databricks/PySpark learning exercise, not a production rides platform. Its workflow generates simulated, deliberately skewed rides and driver/zone dimensions, then demonstrates partitioning, lazy aggregation, join strategies, skew handling, Delta updates, and window metrics. The graph focuses on those product stages; several relationships are supported by the README because notebook excerpts omit intervening cells, so unsampled implementation details are not inferred.
Main entry points: exercise_indrive_spark.py Ride Generation
Components 11
Simulated Data
- Ride Generation
exercise_indrive_spark.py - Driver and Zone Data
exercise_indrive_spark.py
Spark Exercises
- Partition Exercise
exercise_indrive_spark.py - Lazy Aggregation
exercise_indrive_spark.py - Narrow and Wide Ops
exercise_indrive_spark.py - Join Strategies
exercise_indrive_spark.py
Skew Analysis
- Skew Measurement
exercise_indrive_spark.py - Salted Aggregation
exercise_indrive_spark.py
Delta Analytics
- Driver Delta Table
exercise_indrive_spark.py - Window Metrics
exercise_indrive_spark.py
Other components
- Learner
Connections 10
- Learner → Ride Generation: runs notebook
README.md - Ride Generation → Partition Exercise: repartitions
exercise_indrive_spark.py - Ride Generation → Lazy Aggregation: aggregates
exercise_indrive_spark.py - Ride Generation → Narrow and Wide Ops: tests operations
exercise_indrive_spark.py - Ride Generation → Join Strategies: joins with
exercise_indrive_spark.py - Driver and Zone Data → Join Strategies: supplies driver data
exercise_indrive_spark.py - Ride Generation → Skew Measurement: measures skew
exercise_indrive_spark.py - Ride Generation → Salted Aggregation: aggregates with salts
exercise_indrive_spark.py - Driver and Zone Data → Driver Delta Table: writes and merges
exercise_indrive_spark.py - Ride Generation → Window Metrics: feeds earnings metrics
README.md