How nabielnx/eds-jakarta-mobility works
This repository is a Jakarta data warehouse and analytics prototype for air quality, Transjakarta trips, and public-service app reviews. The primary workflow is CSV ingestion into text staging, typed transformation and data-quality checks, then dimensional warehouse loading and dashboard analysis through mart views. It also records ETL runs, quality results, rejected rows, lineage, and a simulated incremental Transjakarta feed. The main pipeline and dashboard calls are visible in excerpts or the source index; SQL schema and mart roles are documented in the README, but their SQL implementations were not sampled.
Main entry points: run_etl.py Full ETL Runner · extract.py CSV Extract · metadata.py Run & Lineage Metadata · db.py Database Access
Components 16
ETL Pipeline
- Full ETL Runner
eds/run_etl.py - Incremental Runner
eds/incremental.py - CSV Extract
eds/extract.py - Domain Transforms
eds/transform.py - Warehouse Loader
eds/load.py - Runtime Configuration
eds/config.py
Quality & Governance
- DQ Rules
eds/quality.py - Run & Lineage Metadata
eds/metadata.py - Governance Store
sql/03_meta.sql
Data Warehouse
- Text Staging
sql/01_staging.sql - Dimensional Warehouse
sql/02_warehouse.sql - Database Access
eds/db.py
Analytics & Dashboard
- Analytical Mart Views
sql/05_mart_views.sql - Streamlit Dashboard
app/dashboard.py
Other components
- Source CSVs
- Analyst / Viewer
Connections 22
- Source CSVs → CSV Extract: provides files
README.md - Full ETL Runner → CSV Extract: invokes extraction
eds/run_etl.py - Full ETL Runner → Domain Transforms: dispatches transforms
eds/run_etl.py - Full ETL Runner → Warehouse Loader: dispatches loads
eds/run_etl.py - Full ETL Runner → Run & Lineage Metadata: records run
eds/run_etl.py - CSV Extract → Text Staging: writes raw rows
eds/extract.py - Full ETL Runner → Text Staging: reads staged rows
eds/run_etl.py - Domain Transforms → DQ Rules: evaluates rules
eds/transform.py - Domain Transforms → Run & Lineage Metadata: supplies DQ and rejects
eds/run_etl.py - Warehouse Loader → Dimensional Warehouse: loads facts and dimensions
README.md - Run & Lineage Metadata → Governance Store: stores run DQ quarantine lineage
README.md - Streamlit Dashboard → Analytical Mart Views: queries analytics views
app/dashboard.py - Analyst / Viewer → Streamlit Dashboard: views insights
README.md - Database Access → Runtime Configuration: uses connection settings
eds/db.py - CSV Extract → Database Access: copies staging rows
eds/extract.py - Warehouse Loader → Database Access: uses COPY helper
eds/load.py - Run & Lineage Metadata → Database Access: uses COPY helper
eds/metadata.py - Streamlit Dashboard → Database Access: queries database
app/dashboard.py - Incremental Runner → CSV Extract: appends daily batch
eds/incremental.py - Incremental Runner → Domain Transforms: dispatches trip transform
eds/incremental.py - Incremental Runner → Warehouse Loader: dispatches trip load
eds/incremental.py - Incremental Runner → Run & Lineage Metadata: records batch results
eds/incremental.py