# ryadulseam/fleetpulse architecture

> Architecture diagram of the GitHub repository ryadulseam/fleetpulse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ryadulseam/fleetpulse
- Repository: https://github.com/ryadulseam/fleetpulse

## Overview

FleetPulse replays historical logistics data as simulated real-time events: it prepares and enriches records, publishes events, ingests them into PostgreSQL, builds analytical warehouse layers, and exposes operational views and predictions. Separate branches train delay/ETA models and forecast demand; the README also documents a data-quality gate. The graph includes shared event, feature, weather, ingestion, and configuration components. Some pipeline and service excerpts were incomplete or unsampled, so their links are limited to README statements and SOURCE INDEX references rather than inferred implementation details.

Main entry points: [`pipeline/03_enrich.py`](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/03_enrich.py) (Enrich records), [`sql`](https://github.com/ryadulseam/fleetpulse/tree/main/sql) (Bronze / Silver / Gold), [`pipeline/04_replay_stream.py`](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/04_replay_stream.py) (Historical replay), [`pipeline/05_consume_to_postgres.py`](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/05_consume_to_postgres.py) (Bronze consumer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_preparation["Data preparation"]
  node_convert["Convert and profile"]
  node_clean["Clean and validate"]
  node_enrich["Enrich records<br/>[03_enrich.py]"]
  node_weather["Weather lookup<br/>[weather.py]"]
  node_config["Config and DB helpers<br/>[common.py]"]
end

subgraph group_streaming["Event streaming"]
  node_replay["Historical replay"]
  node_event_model["Event model<br/>[events.py]"]
  node_broker["Redpanda / Kafka"]
  node_consumer["Bronze consumer"]
  node_ingest["Event validation and writers<br/>[ingest.py]"]
end

subgraph group_warehouse["Warehouse and quality"]
  node_postgres[("PostgreSQL warehouse")]
  node_warehouse_build["Build warehouse layers"]
  node_sql_layers[("Bronze / Silver / Gold")]
  node_dq["Data-quality gate"]
end

subgraph group_analytics["Models and forecasting"]
  node_features["Prediction features<br/>[features.py]"]
  node_model_helpers["Model helpers<br/>[ml.py]"]
  node_train["Delay and ETA training<br/>[08_train_models.py]"]
  node_forecast["Demand forecasting"]
end

subgraph group_serving["User-facing services"]
  node_dashboard["Live dashboard<br/>[app.py]"]
  node_api["Prediction and KPI API<br/>[main.py]"]
end

node_source_data["Historical CSVs"]
node_operator(("Operator / API client"))

node_source_data -->|"reads CSVs"| node_convert
node_convert -->|"prepares inputs"| node_clean
node_clean -->|"provides cleaned data"| node_enrich
node_enrich -->|"loads weather"| node_weather
node_enrich -->|"provides enriched data"| node_replay
node_replay -->|"builds events"| node_event_model
node_replay -->|"publishes events"| node_broker
node_broker -->|"delivers events"| node_consumer
node_consumer -->|"validates and writes"| node_ingest
node_consumer -->|"writes Bronze"| node_postgres
node_postgres -->|"supplies warehouse data"| node_warehouse_build
node_warehouse_build -->|"builds layers"| node_sql_layers
node_sql_layers -->|"stores analytical layers"| node_postgres
node_sql_layers -->|"is checked by"| node_dq
node_dq -->|"gates warehouse"| node_sql_layers
node_enrich -->|"provides training data"| node_train
node_train -->|"builds features"| node_features
node_train -->|"uses model helpers"| node_model_helpers
node_enrich -->|"provides order history"| node_forecast
node_forecast -->|"loads Gold forecast"| node_postgres
node_operator -->|"views operations"| node_dashboard
node_sql_layers -->|"serves dashboard queries"| node_dashboard
node_operator -->|"requests predictions"| node_api
node_api -->|"uses feature schema"| node_features
node_api -->|"reads warehouse KPIs"| node_postgres

click node_source_data "https://github.com/ryadulseam/fleetpulse/tree/main/data"
click node_convert "https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/01_convert_and_profile.py"
click node_clean "https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/02_clean_standardize.py"
click node_enrich "https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/03_enrich.py"
click node_weather "https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/weather.py"
click node_config "https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/common.py"
click node_replay "https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/04_replay_stream.py"
click node_event_model "https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/events.py"
click node_consumer "https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/05_consume_to_postgres.py"
click node_ingest "https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/ingest.py"
click node_warehouse_build "https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/06_build_warehouse.py"
click node_sql_layers "https://github.com/ryadulseam/fleetpulse/tree/main/sql"
click node_dq "https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/07_run_dq_checks.py"
click node_features "https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/features.py"
click node_model_helpers "https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/ml.py"
click node_train "https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/08_train_models.py"
click node_forecast "https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/09_forecast_demand.py"
click node_dashboard "https://github.com/ryadulseam/fleetpulse/blob/main/services/dashboard/app.py"
click node_api "https://github.com/ryadulseam/fleetpulse/blob/main/services/api/main.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_convert,node_clean,node_enrich,node_weather,node_config,node_operator toneBlue
class node_replay,node_event_model,node_broker,node_consumer,node_ingest toneAmber
class node_postgres,node_warehouse_build,node_sql_layers,node_dq toneMint
class node_features,node_model_helpers,node_train,node_forecast toneRose
class node_dashboard,node_api toneIndigo
class node_source_data toneTeal
```

## Components

### Data preparation

- [Convert and profile](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/01_convert_and_profile.py): `pipeline/01_convert_and_profile.py`
- [Clean and validate](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/02_clean_standardize.py): `pipeline/02_clean_standardize.py`
- [Enrich records](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/03_enrich.py): `pipeline/03_enrich.py`
- [Weather lookup](https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/weather.py): `src/fleetpulse/weather.py`
- [Config and DB helpers](https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/common.py): `src/fleetpulse/common.py`

### Event streaming

- [Historical replay](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/04_replay_stream.py): `pipeline/04_replay_stream.py`
- [Event model](https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/events.py): `src/fleetpulse/events.py`
- **Redpanda / Kafka**
- [Bronze consumer](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/05_consume_to_postgres.py): `pipeline/05_consume_to_postgres.py`
- [Event validation and writers](https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/ingest.py): `src/fleetpulse/ingest.py`

### Warehouse and quality

- **PostgreSQL warehouse**
- [Build warehouse layers](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/06_build_warehouse.py): `pipeline/06_build_warehouse.py`
- [Bronze / Silver / Gold](https://github.com/ryadulseam/fleetpulse/tree/main/sql): `sql`
- [Data-quality gate](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/07_run_dq_checks.py): `pipeline/07_run_dq_checks.py`

### Models and forecasting

- [Prediction features](https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/features.py): `src/fleetpulse/features.py`
- [Model helpers](https://github.com/ryadulseam/fleetpulse/blob/main/src/fleetpulse/ml.py): `src/fleetpulse/ml.py`
- [Delay and ETA training](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/08_train_models.py): `pipeline/08_train_models.py`
- [Demand forecasting](https://github.com/ryadulseam/fleetpulse/blob/main/pipeline/09_forecast_demand.py): `pipeline/09_forecast_demand.py`

### User-facing services

- [Live dashboard](https://github.com/ryadulseam/fleetpulse/blob/main/services/dashboard/app.py): `services/dashboard/app.py`
- [Prediction and KPI API](https://github.com/ryadulseam/fleetpulse/blob/main/services/api/main.py): `services/api/main.py`

### Other components

- [Historical CSVs](https://github.com/ryadulseam/fleetpulse/tree/main/data): `data`
- **Operator / API client**

## Connections

- Historical CSVs → Convert and profile: reads CSVs (evidence: `README.md`)
- Convert and profile → Clean and validate: prepares inputs (evidence: `README.md`)
- Clean and validate → Enrich records: provides cleaned data (evidence: `README.md`)
- Enrich records → Weather lookup: loads weather (evidence: `pipeline/03_enrich.py`)
- Enrich records → Historical replay: provides enriched data (evidence: `README.md`)
- Historical replay → Event model: builds events (evidence: `pipeline/04_replay_stream.py`)
- Historical replay → Redpanda / Kafka: publishes events (evidence: `README.md`)
- Redpanda / Kafka → Bronze consumer: delivers events (evidence: `README.md`)
- Bronze consumer → Event validation and writers: validates and writes (evidence: `pipeline/05_consume_to_postgres.py`)
- Bronze consumer → PostgreSQL warehouse: writes Bronze (evidence: `README.md`)
- PostgreSQL warehouse → Build warehouse layers: supplies warehouse data (evidence: `README.md`)
- Build warehouse layers → Bronze / Silver / Gold: builds layers (evidence: `README.md`)
- Bronze / Silver / Gold → PostgreSQL warehouse: stores analytical layers (evidence: `README.md`)
- Bronze / Silver / Gold → Data-quality gate: is checked by (evidence: `README.md`)
- Data-quality gate → Bronze / Silver / Gold: gates warehouse (evidence: `README.md`)
- Enrich records → Delay and ETA training: provides training data (evidence: `README.md`)
- Delay and ETA training → Prediction features: builds features (evidence: `pipeline/08_train_models.py`)
- Delay and ETA training → Model helpers: uses model helpers (evidence: `pipeline/08_train_models.py`)
- Enrich records → Demand forecasting: provides order history (evidence: `README.md`)
- Demand forecasting → PostgreSQL warehouse: loads Gold forecast (evidence: `pipeline/09_forecast_demand.py`)
- Operator / API client → Live dashboard: views operations (evidence: `README.md`)
- Bronze / Silver / Gold → Live dashboard: serves dashboard queries (evidence: `README.md`)
- Operator / API client → Prediction and KPI API: requests predictions (evidence: `services/api/main.py`)
- Prediction and KPI API → Prediction features: uses feature schema (evidence: `services/api/main.py`)
- Prediction and KPI API → PostgreSQL warehouse: reads warehouse KPIs (evidence: `README.md`)