# nabielnx/eds-jakarta-mobility architecture

> Architecture diagram of the GitHub repository nabielnx/eds-jakarta-mobility, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nabielnx/eds-jakarta-mobility
- Repository: https://github.com/nabielnx/eds-jakarta-mobility

## Overview

This repository is a Jakarta data warehouse and analytics prototype for air quality, Transjakarta trips, and public-service app reviews. The primary workflow is CSV ingestion into text staging, typed transformation and data-quality checks, then dimensional warehouse loading and dashboard analysis through mart views. It also records ETL runs, quality results, rejected rows, lineage, and a simulated incremental Transjakarta feed. The main pipeline and dashboard calls are visible in excerpts or the source index; SQL schema and mart roles are documented in the README, but their SQL implementations were not sampled.

Main entry points: [`eds/run_etl.py`](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/run_etl.py) (Full ETL Runner), [`eds/extract.py`](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/extract.py) (CSV Extract), [`eds/metadata.py`](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/metadata.py) (Run & Lineage Metadata), [`eds/db.py`](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/db.py) (Database Access)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["ETL Pipeline"]
  node_full_etl["Full ETL Runner<br/>[run_etl.py]"]
  node_incremental["Incremental Runner<br/>[incremental.py]"]
  node_extract["CSV Extract<br/>[extract.py]"]
  node_transform["Domain Transforms<br/>[transform.py]"]
  node_loader["Warehouse Loader<br/>[load.py]"]
  node_config["Runtime Configuration<br/>[config.py]"]
end

subgraph group_governance["Quality &amp; Governance"]
  node_quality_rules["DQ Rules<br/>[quality.py]"]
  node_metadata["Run &amp; Lineage Metadata<br/>[metadata.py]"]
  node_meta_store[("Governance Store<br/>[03_meta.sql]")]
end

subgraph group_warehouse["Data Warehouse"]
  node_staging[("Text Staging<br/>[01_staging.sql]")]
  node_warehouse_store[("Dimensional Warehouse<br/>[02_warehouse.sql]")]
  node_db_access["Database Access<br/>[db.py]"]
end

subgraph group_analytics["Analytics &amp; Dashboard"]
  node_mart_views[("Analytical Mart Views<br/>[05_mart_views.sql]")]
  node_dashboard["Streamlit Dashboard<br/>[dashboard.py]"]
end

node_csv["Source CSVs"]
node_user(("Analyst / Viewer"))

node_csv -->|"provides files"| node_extract
node_full_etl -->|"invokes extraction"| node_extract
node_full_etl -->|"dispatches transforms"| node_transform
node_full_etl -->|"dispatches loads"| node_loader
node_full_etl -->|"records run"| node_metadata
node_extract -->|"writes raw rows"| node_staging
node_full_etl -->|"reads staged rows"| node_staging
node_transform -->|"evaluates rules"| node_quality_rules
node_transform -->|"supplies DQ and rejects"| node_metadata
node_loader -->|"loads facts and dimensions"| node_warehouse_store
node_metadata -->|"stores run DQ quarantine lineage"| node_meta_store
node_dashboard -->|"queries analytics views"| node_mart_views
node_user -->|"views insights"| node_dashboard
node_db_access -->|"uses connection settings"| node_config
node_extract -->|"copies staging rows"| node_db_access
node_loader -->|"uses COPY helper"| node_db_access
node_metadata -->|"uses COPY helper"| node_db_access
node_dashboard -->|"queries database"| node_db_access
node_incremental -->|"appends daily batch"| node_extract
node_incremental -->|"dispatches trip transform"| node_transform
node_incremental -->|"dispatches trip load"| node_loader
node_incremental -->|"records batch results"| node_metadata

click node_full_etl "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/run_etl.py"
click node_incremental "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/incremental.py"
click node_extract "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/extract.py"
click node_staging "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/sql/01_staging.sql"
click node_transform "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/transform.py"
click node_quality_rules "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/quality.py"
click node_metadata "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/metadata.py"
click node_meta_store "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/sql/03_meta.sql"
click node_loader "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/load.py"
click node_warehouse_store "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/sql/02_warehouse.sql"
click node_mart_views "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/sql/05_mart_views.sql"
click node_dashboard "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/app/dashboard.py"
click node_db_access "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/db.py"
click node_config "https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_full_etl,node_incremental,node_extract,node_transform,node_loader,node_config,node_user toneBlue
class node_quality_rules,node_metadata,node_meta_store toneAmber
class node_staging,node_warehouse_store,node_db_access toneMint
class node_mart_views,node_dashboard toneRose
class node_csv toneIndigo
```

## Components

### ETL Pipeline

- [Full ETL Runner](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/run_etl.py): `eds/run_etl.py`
- [Incremental Runner](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/incremental.py): `eds/incremental.py`
- [CSV Extract](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/extract.py): `eds/extract.py`
- [Domain Transforms](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/transform.py): `eds/transform.py`
- [Warehouse Loader](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/load.py): `eds/load.py`
- [Runtime Configuration](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/config.py): `eds/config.py`

### Quality & Governance

- [DQ Rules](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/quality.py): `eds/quality.py`
- [Run & Lineage Metadata](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/metadata.py): `eds/metadata.py`
- [Governance Store](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/sql/03_meta.sql): `sql/03_meta.sql`

### Data Warehouse

- [Text Staging](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/sql/01_staging.sql): `sql/01_staging.sql`
- [Dimensional Warehouse](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/sql/02_warehouse.sql): `sql/02_warehouse.sql`
- [Database Access](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/eds/db.py): `eds/db.py`

### Analytics & Dashboard

- [Analytical Mart Views](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/sql/05_mart_views.sql): `sql/05_mart_views.sql`
- [Streamlit Dashboard](https://github.com/nabielnx/eds-jakarta-mobility/blob/main/app/dashboard.py): `app/dashboard.py`

### Other components

- **Source CSVs**
- **Analyst / Viewer**

## Connections

- Source CSVs → CSV Extract: provides files (evidence: `README.md`)
- Full ETL Runner → CSV Extract: invokes extraction (evidence: `eds/run_etl.py`)
- Full ETL Runner → Domain Transforms: dispatches transforms (evidence: `eds/run_etl.py`)
- Full ETL Runner → Warehouse Loader: dispatches loads (evidence: `eds/run_etl.py`)
- Full ETL Runner → Run & Lineage Metadata: records run (evidence: `eds/run_etl.py`)
- CSV Extract → Text Staging: writes raw rows (evidence: `eds/extract.py`)
- Full ETL Runner → Text Staging: reads staged rows (evidence: `eds/run_etl.py`)
- Domain Transforms → DQ Rules: evaluates rules (evidence: `eds/transform.py`)
- Domain Transforms → Run & Lineage Metadata: supplies DQ and rejects (evidence: `eds/run_etl.py`)
- Warehouse Loader → Dimensional Warehouse: loads facts and dimensions (evidence: `README.md`)
- Run & Lineage Metadata → Governance Store: stores run DQ quarantine lineage (evidence: `README.md`)
- Streamlit Dashboard → Analytical Mart Views: queries analytics views (evidence: `app/dashboard.py`)
- Analyst / Viewer → Streamlit Dashboard: views insights (evidence: `README.md`)
- Database Access → Runtime Configuration: uses connection settings (evidence: `eds/db.py`)
- CSV Extract → Database Access: copies staging rows (evidence: `eds/extract.py`)
- Warehouse Loader → Database Access: uses COPY helper (evidence: `eds/load.py`)
- Run & Lineage Metadata → Database Access: uses COPY helper (evidence: `eds/metadata.py`)
- Streamlit Dashboard → Database Access: queries database (evidence: `app/dashboard.py`)
- Incremental Runner → CSV Extract: appends daily batch (evidence: `eds/incremental.py`)
- Incremental Runner → Domain Transforms: dispatches trip transform (evidence: `eds/incremental.py`)
- Incremental Runner → Warehouse Loader: dispatches trip load (evidence: `eds/incremental.py`)
- Incremental Runner → Run & Lineage Metadata: records batch results (evidence: `eds/incremental.py`)
