How daerenkim/retail-etl-pipeline works
Aangan Retail’s product is an incremental, idempotent retail-sales ETL pipeline, with a downstream dbt analytics layer and Streamlit dashboard. Its primary workflow discovers and fingerprints landing CSVs, maps their headers, standardises and validates rows, loads warehouse partitions and Parquet, then reconciles totals and reports quality issues. The graph includes the documented analytics and alerting branches; unsampled subsystem links rely on README descriptions rather than inspected implementation.
Main entry points: pipeline.py Pipeline orchestration · cli.py CLI commands · load.py Dimension and partition load · config.py Settings and contract
Components 21
Run and Intake
- CLI commands
etl/cli.py - Settings and contract
etl/config.py - Landing CSV files
- File discovery
etl/discover.py - File fingerprinting
etl/fingerprint.py - Pipeline orchestration
etl/pipeline.py
Data Processing
- CSV extraction
etl/extract.py - Schema mapping
etl/schema.py - Data standardisation
etl/transform.py - Row validation
etl/validate.py
Warehouse and Quality
- Warehouse adapter
etl/warehouse.py - SQLite or PostgreSQL
- Dimension and partition load
etl/load.py - Parquet staging
- Missing-store alerts
etl/alerts.py - Reconciliation and reports
etl/reconcile.py - Structured run logging
etl/logging_utils.py
Analytics and Dashboard
- dbt staging models
analytics/models/staging - dbt marts
analytics/models/marts - Streamlit dashboard
dashboard/app.py
Other components
- Pipeline operator
Connections 24
- Pipeline operator → CLI commands: runs commands
README.md - CLI commands → Settings and contract: loads settings
etl/cli.py - CLI commands → Pipeline orchestration: starts run
etl/cli.py - Landing CSV files → File discovery: provides CSVs
README.md - Settings and contract → Pipeline orchestration: configures run
README.md - Pipeline orchestration → File discovery: discovers files
etl/pipeline.py - Pipeline orchestration → File fingerprinting: fingerprints files
etl/pipeline.py - Pipeline orchestration → CSV extraction: extracts rows
etl/pipeline.py - Pipeline orchestration → Schema mapping: maps headers
etl/pipeline.py - Pipeline orchestration → Data standardisation: standardises rows
etl/pipeline.py - Pipeline orchestration → Row validation: validates rows
etl/pipeline.py - Pipeline orchestration → Dimension and partition load: loads partition
etl/pipeline.py - Pipeline orchestration → Missing-store alerts: updates alerts
etl/pipeline.py - Pipeline orchestration → Reconciliation and reports: builds reports
etl/pipeline.py - Pipeline orchestration → Structured run logging: records events
etl/pipeline.py - Dimension and partition load → Warehouse adapter: writes data
etl/load.py - Warehouse adapter → SQLite or PostgreSQL: persists records
README.md - Dimension and partition load → Parquet staging: writes Parquet
README.md - Row validation → SQLite or PostgreSQL: stores rejected rows
README.md - Missing-store alerts → SQLite or PostgreSQL: reads and updates alerts
etl/alerts.py - Reconciliation and reports → SQLite or PostgreSQL: reads loaded totals
etl/reconcile.py - Parquet staging → dbt staging models: feeds staging
README.md - dbt staging models → dbt marts: models analytics
README.md - dbt marts → Streamlit dashboard: feeds dashboard
README.md