# amtorigami/indrive_spark architecture

> Architecture diagram of the GitHub repository amtorigami/indrive_spark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amtorigami/indrive_spark
- Repository: https://github.com/amtorigami/indrive_spark

## Overview

This repository is a Databricks/PySpark learning exercise, not a production rides platform. Its workflow generates simulated, deliberately skewed rides and driver/zone dimensions, then demonstrates partitioning, lazy aggregation, join strategies, skew handling, Delta updates, and window metrics. The graph focuses on those product stages; several relationships are supported by the README because notebook excerpts omit intervening cells, so unsampled implementation details are not inferred.

Main entry points: [`exercise_indrive_spark.py`](https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py) (Ride Generation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Simulated Data"]
  node_rides["Ride Generation"]
  node_dimensions["Driver and Zone Data"]
end

subgraph group_spark["Spark Exercises"]
  node_partitions["Partition Exercise"]
  node_lazy_plan["Lazy Aggregation"]
  node_operations["Narrow and Wide Ops"]
  node_joins["Join Strategies"]
end

subgraph group_skew["Skew Analysis"]
  node_skew_measure["Skew Measurement"]
  node_salting["Salted Aggregation"]
end

subgraph group_delta["Delta Analytics"]
  node_delta_table[("Driver Delta Table")]
  node_running_metrics["Window Metrics"]
end

node_learner(("Learner"))

node_learner -->|"runs notebook"| node_rides
node_rides -->|"repartitions"| node_partitions
node_rides -->|"aggregates"| node_lazy_plan
node_rides -->|"tests operations"| node_operations
node_rides -->|"joins with"| node_joins
node_dimensions -->|"supplies driver data"| node_joins
node_rides -->|"measures skew"| node_skew_measure
node_rides -->|"aggregates with salts"| node_salting
node_dimensions -->|"writes and merges"| node_delta_table
node_rides -.->|"feeds earnings metrics"| node_running_metrics

click node_rides "https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py"
click node_dimensions "https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py"
click node_partitions "https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py"
click node_lazy_plan "https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py"
click node_operations "https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py"
click node_joins "https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py"
click node_skew_measure "https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py"
click node_salting "https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py"
click node_delta_table "https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py"
click node_running_metrics "https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_rides,node_dimensions toneBlue
class node_partitions,node_lazy_plan,node_operations,node_joins toneAmber
class node_skew_measure,node_salting toneMint
class node_delta_table,node_running_metrics toneRose
class node_learner toneIndigo
```

## Components

### Simulated Data

- [Ride Generation](https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py): `exercise_indrive_spark.py`
- [Driver and Zone Data](https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py): `exercise_indrive_spark.py`

### Spark Exercises

- [Partition Exercise](https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py): `exercise_indrive_spark.py`
- [Lazy Aggregation](https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py): `exercise_indrive_spark.py`
- [Narrow and Wide Ops](https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py): `exercise_indrive_spark.py`
- [Join Strategies](https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py): `exercise_indrive_spark.py`

### Skew Analysis

- [Skew Measurement](https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py): `exercise_indrive_spark.py`
- [Salted Aggregation](https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py): `exercise_indrive_spark.py`

### Delta Analytics

- [Driver Delta Table](https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py): `exercise_indrive_spark.py`
- [Window Metrics](https://github.com/amtorigami/indrive_spark/blob/main/exercise_indrive_spark.py): `exercise_indrive_spark.py`

### Other components

- **Learner**

## Connections

- Learner → Ride Generation: runs notebook (evidence: `README.md`)
- Ride Generation → Partition Exercise: repartitions (evidence: `exercise_indrive_spark.py`)
- Ride Generation → Lazy Aggregation: aggregates (evidence: `exercise_indrive_spark.py`)
- Ride Generation → Narrow and Wide Ops: tests operations (evidence: `exercise_indrive_spark.py`)
- Ride Generation → Join Strategies: joins with (evidence: `exercise_indrive_spark.py`)
- Driver and Zone Data → Join Strategies: supplies driver data (evidence: `exercise_indrive_spark.py`)
- Ride Generation → Skew Measurement: measures skew (evidence: `exercise_indrive_spark.py`)
- Ride Generation → Salted Aggregation: aggregates with salts (evidence: `exercise_indrive_spark.py`)
- Driver and Zone Data → Driver Delta Table: writes and merges (evidence: `exercise_indrive_spark.py`)
- Ride Generation → Window Metrics: feeds earnings metrics (evidence: `README.md`)
