How khprateek/enterprise-supply-chain-inventory-optimization-hub works
This repository is an analytical supply-chain data platform: it generates operational data, transforms it into a dimensional warehouse, and exposes analytics through Power BI. The principal workflow runs from generated Parquet data through Airflow orchestration and dbt models to the semantic model and dashboards. The tree also documents an optional PySpark/Dataproc path; sampled files show its session and BigQuery I/O utilities, but not substantive Spark transformation jobs, so those are not represented as a processing stage.
Main entry points: orchestrator.py Generation orchestrator · dags Airflow DAGs · marts Dimensional marts · staging Staging models
Components 21
Data generation
- Generation orchestrator
python/orchestrator.py - Dimensions
python/generate_dimensions.py - Procurement
python/generate_procurement.py - Sales fulfillment
python/generate_sales.py - Operational facts
python - Raw Parquet
data/raw
ELT orchestration
- Airflow DAGs
orchestration/airflow/dags
Warehouse modeling
- Staging models
dbt/models/staging - Intermediate metrics
dbt/models/intermediate - Dimensional marts
dbt/models/marts - Product SCD2
dbt/snapshots/snap_dim_product_scd2.sql
Spark integration
- Spark session
pyspark_jobs/utils/spark_session.py - BigQuery I/O
pyspark_jobs/utils/bigquery_io.py
Power BI analytics
- Semantic model
powerbi/model_definitions/model_schema.json - DAX measures
powerbi/measures - Analytics pages
docs/powerbi
Other components
- Source systems
- Cloud Storage
- BigQuery warehouse
- Planners and executives
- Power BI service
Connections 22
- Source systems → Generation orchestrator: simulates inputs
- Generation orchestrator → Dimensions: generates
- Generation orchestrator → Procurement: generates
- Generation orchestrator → Sales fulfillment: generates
- Generation orchestrator → Operational facts: generates
- Generation orchestrator → Raw Parquet: persists Parquet
- Airflow DAGs → Cloud Storage: checks landing
- Airflow DAGs → BigQuery warehouse: syncs partitions
- Airflow DAGs → Staging models: runs dbt
- Staging models → Intermediate metrics: feeds models
- Staging models → Dimensional marts: feeds models
- Intermediate metrics → Dimensional marts: feeds metrics
- Product SCD2 → Dimensional marts: tracks product history
- Dimensional marts → BigQuery warehouse: materializes in
- Airflow DAGs → Power BI service: triggers refresh
- BigQuery warehouse → Semantic model: provides warehouse data
- Semantic model → DAX measures: defines calculations
- Semantic model → Analytics pages: supports pages
- Planners and executives → Analytics pages: explores analytics
- Airflow DAGs → Spark session: submits Spark jobs
- Spark session → BigQuery I/O: enables BigQuery access
- BigQuery I/O → BigQuery warehouse: reads and writes