How tar7nic/flowmetrics works
FlowMetrics turns DataCo supply-chain records into a SQLite star-schema warehouse, KPI analyses, and dashboards. The documented batch workflow explores and cleans source data, transforms it with Spark, loads the warehouse, then supports SQL KPI computation and a Streamlit app. Separate branches export warehouse tables for Power BI and simulate scheduled daily updates with run logging. Edges reflect README phase descriptions and shown calls; unsampled integrations are not inferred.
Main entry points: supply_chain.db SQLite Warehouse · daily_pipeline.py Daily Update Pipeline · clean.py Data Cleaning · DataCoSupplyChainDataset.csv DataCo Dataset
Components 15
Data Preparation
- Data Exploration
etl/explore.py - Data Cleaning
etl/clean.py - Cleaned Domain CSVs
data/raw/cleaned - Spark Transform
etl/spark_transform.py
Warehouse Analytics
- Star Schema Loader
warehouse/warehouse.py - SQLite Warehouse
warehouse/supply_chain.db - SQL KPI Analysis
kpis/kpis.py
Dashboards
- Streamlit Dashboard
dashboard/app.py - Power BI Export
pipelines/powerbi_export.py - Power BI Dashboard
powerbi/data/dash.pbix
Pipeline Operations
- Daily Scheduler
pipelines/scheduler.py - Daily Update Pipeline
pipelines/daily_pipeline.py - Pipeline Run Log
pipelines/pipeline_logs/pipeline_log.db - Run History Viewer
pipelines/view_logs.py
Other components
- DataCo Dataset
data/raw/DataCoSupplyChainDataset.csv
Connections 16
- DataCo Dataset → Data Exploration: inspects
readme.md - DataCo Dataset → Data Cleaning: loads raw data
readme.md - Data Exploration → Data Cleaning: precedes
readme.md - Data Cleaning → Cleaned Domain CSVs: writes CSVs
readme.md - Cleaned Domain CSVs → Spark Transform: reads inputs
readme.md - Spark Transform → Star Schema Loader: provides processed data
readme.md - Star Schema Loader → SQLite Warehouse: loads schema
warehouse/warehouse.py - SQLite Warehouse → SQL KPI Analysis: queries
readme.md - SQLite Warehouse → Streamlit Dashboard: reads data
dashboard/app.py - SQLite Warehouse → Power BI Export: exports tables
pipelines/powerbi_export.py - Power BI Export → Power BI Dashboard: supplies CSVs
readme.md - Daily Scheduler → Daily Update Pipeline: launches
pipelines/scheduler.py - Daily Update Pipeline → SQLite Warehouse: inserts orders
pipelines/daily_pipeline.py - Daily Update Pipeline → Pipeline Run Log: records runs
pipelines/daily_pipeline.py - Daily Update Pipeline → Power BI Dashboard: re-exports tables
pipelines/daily_pipeline.py - Run History Viewer → Pipeline Run Log: reads history
pipelines/view_logs.py