How khprateek/supply-chain-inventory-optimization-hub works
This repository is an end-to-end supply-chain analytics platform: it generates operational datasets, transforms them into a dimensional warehouse, and exposes measures and dashboards for planners and executives. The principal documented workflow is daily Airflow orchestration from raw-file arrival through BigQuery and dbt to Power BI refresh. A separate PySpark processing path and local Polars marts are also represented. The graph follows the README and sampled files; several dashboard and warehouse capabilities are documented but their runtime implementation was not sampled.
Main entry points: dag_supply_chain_daily_elt.py Daily ELT DAG · orchestrator.py Dataset orchestrator · model_schema.json Power BI semantic model · staging Staging models
Components 24
Data generation
- Dataset orchestrator
python/orchestrator.py - Conformed dimensions
python/generate_dimensions.py - Operational datasets
python - Raw Parquet files
data/raw - Incremental state
python/incremental_state.py
Warehouse modeling
- BigQuery raw
dbt/models/sources.yml - Staging models
dbt/models/staging - Intermediate metrics
dbt/models/intermediate - Dimensional marts
dbt/models/marts - Product SCD2
dbt/snapshots/snap_dim_product_scd2.sql - Data quality tests
dbt/tests - Analytical warehouse
- PySpark staging
pyspark_jobs/complex_transforms/01_staging.py - PySpark fact builds
pyspark_jobs/complex_transforms/03_fact_tables.py - Spark BigQuery I/O
pyspark_jobs/utils/bigquery_io.py - Local Polars marts
polars_jobs/01_build_marts.py
Pipeline orchestration
- Daily ELT DAG
orchestration/airflow/dags/dag_supply_chain_daily_elt.py - GCS landing
Analytics and reporting
- Power BI semantic model
powerbi/model_definitions/model_schema.json - DAX measure libraries
powerbi/measures - Dynamic row security
docs/powerbi/rls_design.md - Report pages
docs/powerbi - Power BI service
Other components
- Supply-chain users
Connections 24
- Dataset orchestrator → Conformed dimensions: generates
- Dataset orchestrator → Operational datasets: generates
- Dataset orchestrator → Incremental state: reads and updates
- Dataset orchestrator → Raw Parquet files: writes
- Daily ELT DAG → GCS landing: checks arrival
- GCS landing → Daily ELT DAG: signals readiness
- Daily ELT DAG → BigQuery raw: syncs partitions
- BigQuery raw → Staging models: provides sources
- Staging models → Intermediate metrics: feeds models
- Intermediate metrics → Dimensional marts: feeds models
- Dimensional marts → Analytical warehouse: materializes
- Staging models → Product SCD2: snapshots products
- Daily ELT DAG → Data quality tests: runs tests
- Data quality tests → Dimensional marts: validates
- Daily ELT DAG → Power BI service: requests refresh
- Analytical warehouse → Power BI semantic model: provides warehouse data
- DAX measure libraries → Power BI semantic model: defines calculations
- Dynamic row security → Power BI semantic model: specifies access rules
- Power BI semantic model → Report pages: supports reports
- Report pages → Supply-chain users: presents analytics
- PySpark staging → PySpark fact builds: stages inputs for
- PySpark fact builds → Spark BigQuery I/O: uses read/write helpers
- Spark BigQuery I/O → Analytical warehouse: reads and writes
- Local Polars marts → Supply-chain users: produces local marts