# daerenkim/retail-etl-pipeline architecture

> Architecture diagram of the GitHub repository daerenkim/retail-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/daerenkim/retail-etl-pipeline
- Repository: https://github.com/daerenkim/retail-etl-pipeline

## Overview

Aangan Retail’s product is an incremental, idempotent retail-sales ETL pipeline, with a downstream dbt analytics layer and Streamlit dashboard. Its primary workflow discovers and fingerprints landing CSVs, maps their headers, standardises and validates rows, loads warehouse partitions and Parquet, then reconciles totals and reports quality issues. The graph includes the documented analytics and alerting branches; unsampled subsystem links rely on README descriptions rather than inspected implementation.

Main entry points: [`etl/pipeline.py`](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/pipeline.py) (Pipeline orchestration), [`etl/cli.py`](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/cli.py) (CLI commands), [`etl/load.py`](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/load.py) (Dimension and partition load), [`etl/config.py`](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/config.py) (Settings and contract)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_intake["Run and Intake"]
  node_cli["CLI commands<br/>[cli.py]"]
  node_config["Settings and contract<br/>[config.py]"]
  node_landing[("Landing CSV files")]
  node_discovery["File discovery<br/>[discover.py]"]
  node_fingerprint["File fingerprinting<br/>[fingerprint.py]"]
  node_orchestrator["Pipeline orchestration<br/>[pipeline.py]"]
end

subgraph group_processing["Data Processing"]
  node_extraction["CSV extraction<br/>[extract.py]"]
  node_schema_mapping["Schema mapping<br/>[schema.py]"]
  node_standardisation["Data standardisation<br/>[transform.py]"]
  node_validation["Row validation<br/>[validate.py]"]
end

subgraph group_warehouse_quality["Warehouse and Quality"]
  node_warehouse_adapter["Warehouse adapter<br/>[warehouse.py]"]
  node_warehouse[("SQLite or PostgreSQL")]
  node_loading["Dimension and partition load<br/>[load.py]"]
  node_silver[("Parquet staging")]
  node_alerts["Missing-store alerts<br/>[alerts.py]"]
  node_reconciliation["Reconciliation and reports<br/>[reconcile.py]"]
  node_logging["Structured run logging<br/>[logging_utils.py]"]
end

subgraph group_analytics["Analytics and Dashboard"]
  node_dbt_staging["dbt staging models"]
  node_dbt_marts["dbt marts"]
  node_dashboard["Streamlit dashboard<br/>[app.py]"]
end

node_operator(("Pipeline operator"))

node_operator -->|"runs commands"| node_cli
node_cli -->|"loads settings"| node_config
node_cli -->|"starts run"| node_orchestrator
node_landing -->|"provides CSVs"| node_discovery
node_config -->|"configures run"| node_orchestrator
node_orchestrator -->|"discovers files"| node_discovery
node_orchestrator -->|"fingerprints files"| node_fingerprint
node_orchestrator -->|"extracts rows"| node_extraction
node_orchestrator -->|"maps headers"| node_schema_mapping
node_orchestrator -->|"standardises rows"| node_standardisation
node_orchestrator -->|"validates rows"| node_validation
node_orchestrator -->|"loads partition"| node_loading
node_orchestrator -->|"updates alerts"| node_alerts
node_orchestrator -->|"builds reports"| node_reconciliation
node_orchestrator -->|"records events"| node_logging
node_loading -->|"writes data"| node_warehouse_adapter
node_warehouse_adapter -->|"persists records"| node_warehouse
node_loading -->|"writes Parquet"| node_silver
node_validation -->|"stores rejected rows"| node_warehouse
node_alerts -->|"reads and updates alerts"| node_warehouse
node_reconciliation -->|"reads loaded totals"| node_warehouse
node_silver -->|"feeds staging"| node_dbt_staging
node_dbt_staging -->|"models analytics"| node_dbt_marts
node_dbt_marts -->|"feeds dashboard"| node_dashboard

click node_cli "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/cli.py"
click node_config "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/config.py"
click node_discovery "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/discover.py"
click node_fingerprint "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/fingerprint.py"
click node_orchestrator "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/pipeline.py"
click node_extraction "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/extract.py"
click node_schema_mapping "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/schema.py"
click node_standardisation "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/transform.py"
click node_validation "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/validate.py"
click node_warehouse_adapter "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/warehouse.py"
click node_loading "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/load.py"
click node_alerts "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/alerts.py"
click node_reconciliation "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/reconcile.py"
click node_logging "https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/logging_utils.py"
click node_dbt_staging "https://github.com/daerenkim/retail-etl-pipeline/tree/main/analytics/models/staging"
click node_dbt_marts "https://github.com/daerenkim/retail-etl-pipeline/tree/main/analytics/models/marts"
click node_dashboard "https://github.com/daerenkim/retail-etl-pipeline/blob/main/dashboard/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config,node_landing,node_discovery,node_fingerprint,node_orchestrator toneBlue
class node_extraction,node_schema_mapping,node_standardisation,node_validation toneAmber
class node_warehouse_adapter,node_warehouse,node_loading,node_silver,node_alerts,node_reconciliation,node_logging toneMint
class node_dbt_staging,node_dbt_marts,node_dashboard toneRose
class node_operator toneIndigo
```

## Components

### Run and Intake

- [CLI commands](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/cli.py): `etl/cli.py`
- [Settings and contract](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/config.py): `etl/config.py`
- **Landing CSV files**
- [File discovery](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/discover.py): `etl/discover.py`
- [File fingerprinting](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/fingerprint.py): `etl/fingerprint.py`
- [Pipeline orchestration](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/pipeline.py): `etl/pipeline.py`

### Data Processing

- [CSV extraction](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/extract.py): `etl/extract.py`
- [Schema mapping](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/schema.py): `etl/schema.py`
- [Data standardisation](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/transform.py): `etl/transform.py`
- [Row validation](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/validate.py): `etl/validate.py`

### Warehouse and Quality

- [Warehouse adapter](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/warehouse.py): `etl/warehouse.py`
- **SQLite or PostgreSQL**
- [Dimension and partition load](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/load.py): `etl/load.py`
- **Parquet staging**
- [Missing-store alerts](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/alerts.py): `etl/alerts.py`
- [Reconciliation and reports](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/reconcile.py): `etl/reconcile.py`
- [Structured run logging](https://github.com/daerenkim/retail-etl-pipeline/blob/main/etl/logging_utils.py): `etl/logging_utils.py`

### Analytics and Dashboard

- [dbt staging models](https://github.com/daerenkim/retail-etl-pipeline/tree/main/analytics/models/staging): `analytics/models/staging`
- [dbt marts](https://github.com/daerenkim/retail-etl-pipeline/tree/main/analytics/models/marts): `analytics/models/marts`
- [Streamlit dashboard](https://github.com/daerenkim/retail-etl-pipeline/blob/main/dashboard/app.py): `dashboard/app.py`

### Other components

- **Pipeline operator**

## Connections

- Pipeline operator → CLI commands: runs commands (evidence: `README.md`)
- CLI commands → Settings and contract: loads settings (evidence: `etl/cli.py`)
- CLI commands → Pipeline orchestration: starts run (evidence: `etl/cli.py`)
- Landing CSV files → File discovery: provides CSVs (evidence: `README.md`)
- Settings and contract → Pipeline orchestration: configures run (evidence: `README.md`)
- Pipeline orchestration → File discovery: discovers files (evidence: `etl/pipeline.py`)
- Pipeline orchestration → File fingerprinting: fingerprints files (evidence: `etl/pipeline.py`)
- Pipeline orchestration → CSV extraction: extracts rows (evidence: `etl/pipeline.py`)
- Pipeline orchestration → Schema mapping: maps headers (evidence: `etl/pipeline.py`)
- Pipeline orchestration → Data standardisation: standardises rows (evidence: `etl/pipeline.py`)
- Pipeline orchestration → Row validation: validates rows (evidence: `etl/pipeline.py`)
- Pipeline orchestration → Dimension and partition load: loads partition (evidence: `etl/pipeline.py`)
- Pipeline orchestration → Missing-store alerts: updates alerts (evidence: `etl/pipeline.py`)
- Pipeline orchestration → Reconciliation and reports: builds reports (evidence: `etl/pipeline.py`)
- Pipeline orchestration → Structured run logging: records events (evidence: `etl/pipeline.py`)
- Dimension and partition load → Warehouse adapter: writes data (evidence: `etl/load.py`)
- Warehouse adapter → SQLite or PostgreSQL: persists records (evidence: `README.md`)
- Dimension and partition load → Parquet staging: writes Parquet (evidence: `README.md`)
- Row validation → SQLite or PostgreSQL: stores rejected rows (evidence: `README.md`)
- Missing-store alerts → SQLite or PostgreSQL: reads and updates alerts (evidence: `etl/alerts.py`)
- Reconciliation and reports → SQLite or PostgreSQL: reads loaded totals (evidence: `etl/reconcile.py`)
- Parquet staging → dbt staging models: feeds staging (evidence: `README.md`)
- dbt staging models → dbt marts: models analytics (evidence: `README.md`)
- dbt marts → Streamlit dashboard: feeds dashboard (evidence: `README.md`)
