# tar7nic/flowmetrics architecture

> Architecture diagram of the GitHub repository tar7nic/flowmetrics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tar7nic/flowmetrics
- Repository: https://github.com/tar7nic/flowmetrics

## Overview

FlowMetrics turns DataCo supply-chain records into a SQLite star-schema warehouse, KPI analyses, and dashboards. The documented batch workflow explores and cleans source data, transforms it with Spark, loads the warehouse, then supports SQL KPI computation and a Streamlit app. Separate branches export warehouse tables for Power BI and simulate scheduled daily updates with run logging. Edges reflect README phase descriptions and shown calls; unsampled integrations are not inferred.

Main entry points: [`warehouse/supply_chain.db`](https://github.com/tar7nic/flowmetrics/blob/main/warehouse/supply_chain.db) (SQLite Warehouse), [`pipelines/daily_pipeline.py`](https://github.com/tar7nic/flowmetrics/blob/main/pipelines/daily_pipeline.py) (Daily Update Pipeline), [`etl/clean.py`](https://github.com/tar7nic/flowmetrics/blob/main/etl/clean.py) (Data Cleaning), [`data/raw/DataCoSupplyChainDataset.csv`](https://github.com/tar7nic/flowmetrics/blob/main/data/raw/DataCoSupplyChainDataset.csv) (DataCo Dataset)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data Preparation"]
  node_explore["Data Exploration<br/>[explore.py]"]
  node_clean["Data Cleaning<br/>[clean.py]"]
  node_cleaned["Cleaned Domain CSVs"]
  node_spark["Spark Transform<br/>[spark_transform.py]"]
end

subgraph group_analytics["Warehouse Analytics"]
  node_warehouse_loader["Star Schema Loader<br/>[warehouse.py]"]
  node_warehouse_db[("SQLite Warehouse<br/>[supply_chain.db]")]
  node_kpis["SQL KPI Analysis<br/>[kpis.py]"]
end

subgraph group_presentation["Dashboards"]
  node_streamlit["Streamlit Dashboard<br/>[app.py]"]
  node_exporter["Power BI Export<br/>[powerbi_export.py]"]
  node_powerbi["Power BI Dashboard<br/>[dash.pbix]"]
end

subgraph group_operations["Pipeline Operations"]
  node_scheduler["Daily Scheduler<br/>[scheduler.py]"]
  node_daily["Daily Update Pipeline<br/>[daily_pipeline.py]"]
  node_runlogs[("Pipeline Run Log<br/>[pipeline_log.db]")]
  node_logviewer["Run History Viewer<br/>[view_logs.py]"]
end

node_dataset["DataCo Dataset"]

node_dataset -->|"inspects"| node_explore
node_dataset -->|"loads raw data"| node_clean
node_explore -->|"precedes"| node_clean
node_clean -->|"writes CSVs"| node_cleaned
node_cleaned -->|"reads inputs"| node_spark
node_spark -->|"provides processed data"| node_warehouse_loader
node_warehouse_loader -->|"loads schema"| node_warehouse_db
node_warehouse_db -->|"queries"| node_kpis
node_warehouse_db -->|"reads data"| node_streamlit
node_warehouse_db -->|"exports tables"| node_exporter
node_exporter -->|"supplies CSVs"| node_powerbi
node_scheduler -->|"launches"| node_daily
node_daily -->|"inserts orders"| node_warehouse_db
node_daily -->|"records runs"| node_runlogs
node_daily -->|"re-exports tables"| node_powerbi
node_logviewer -->|"reads history"| node_runlogs

click node_dataset "https://github.com/tar7nic/flowmetrics/blob/main/data/raw/DataCoSupplyChainDataset.csv"
click node_explore "https://github.com/tar7nic/flowmetrics/blob/main/etl/explore.py"
click node_clean "https://github.com/tar7nic/flowmetrics/blob/main/etl/clean.py"
click node_cleaned "https://github.com/tar7nic/flowmetrics/tree/main/data/raw/cleaned"
click node_spark "https://github.com/tar7nic/flowmetrics/blob/main/etl/spark_transform.py"
click node_warehouse_loader "https://github.com/tar7nic/flowmetrics/blob/main/warehouse/warehouse.py"
click node_warehouse_db "https://github.com/tar7nic/flowmetrics/blob/main/warehouse/supply_chain.db"
click node_kpis "https://github.com/tar7nic/flowmetrics/blob/main/kpis/kpis.py"
click node_streamlit "https://github.com/tar7nic/flowmetrics/blob/main/dashboard/app.py"
click node_exporter "https://github.com/tar7nic/flowmetrics/blob/main/pipelines/powerbi_export.py"
click node_powerbi "https://github.com/tar7nic/flowmetrics/blob/main/powerbi/data/dash.pbix"
click node_scheduler "https://github.com/tar7nic/flowmetrics/blob/main/pipelines/scheduler.py"
click node_daily "https://github.com/tar7nic/flowmetrics/blob/main/pipelines/daily_pipeline.py"
click node_runlogs "https://github.com/tar7nic/flowmetrics/blob/main/pipelines/pipeline_logs/pipeline_log.db"
click node_logviewer "https://github.com/tar7nic/flowmetrics/blob/main/pipelines/view_logs.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_explore,node_clean,node_cleaned,node_spark toneBlue
class node_warehouse_loader,node_warehouse_db,node_kpis toneAmber
class node_streamlit,node_exporter,node_powerbi toneMint
class node_scheduler,node_daily,node_runlogs,node_logviewer toneRose
class node_dataset toneTeal
```

## Components

### Data Preparation

- [Data Exploration](https://github.com/tar7nic/flowmetrics/blob/main/etl/explore.py): `etl/explore.py`
- [Data Cleaning](https://github.com/tar7nic/flowmetrics/blob/main/etl/clean.py): `etl/clean.py`
- [Cleaned Domain CSVs](https://github.com/tar7nic/flowmetrics/tree/main/data/raw/cleaned): `data/raw/cleaned`
- [Spark Transform](https://github.com/tar7nic/flowmetrics/blob/main/etl/spark_transform.py): `etl/spark_transform.py`

### Warehouse Analytics

- [Star Schema Loader](https://github.com/tar7nic/flowmetrics/blob/main/warehouse/warehouse.py): `warehouse/warehouse.py`
- [SQLite Warehouse](https://github.com/tar7nic/flowmetrics/blob/main/warehouse/supply_chain.db): `warehouse/supply_chain.db`
- [SQL KPI Analysis](https://github.com/tar7nic/flowmetrics/blob/main/kpis/kpis.py): `kpis/kpis.py`

### Dashboards

- [Streamlit Dashboard](https://github.com/tar7nic/flowmetrics/blob/main/dashboard/app.py): `dashboard/app.py`
- [Power BI Export](https://github.com/tar7nic/flowmetrics/blob/main/pipelines/powerbi_export.py): `pipelines/powerbi_export.py`
- [Power BI Dashboard](https://github.com/tar7nic/flowmetrics/blob/main/powerbi/data/dash.pbix): `powerbi/data/dash.pbix`

### Pipeline Operations

- [Daily Scheduler](https://github.com/tar7nic/flowmetrics/blob/main/pipelines/scheduler.py): `pipelines/scheduler.py`
- [Daily Update Pipeline](https://github.com/tar7nic/flowmetrics/blob/main/pipelines/daily_pipeline.py): `pipelines/daily_pipeline.py`
- [Pipeline Run Log](https://github.com/tar7nic/flowmetrics/blob/main/pipelines/pipeline_logs/pipeline_log.db): `pipelines/pipeline_logs/pipeline_log.db`
- [Run History Viewer](https://github.com/tar7nic/flowmetrics/blob/main/pipelines/view_logs.py): `pipelines/view_logs.py`

### Other components

- [DataCo Dataset](https://github.com/tar7nic/flowmetrics/blob/main/data/raw/DataCoSupplyChainDataset.csv): `data/raw/DataCoSupplyChainDataset.csv`

## Connections

- DataCo Dataset → Data Exploration: inspects (evidence: `readme.md`)
- DataCo Dataset → Data Cleaning: loads raw data (evidence: `readme.md`)
- Data Exploration → Data Cleaning: precedes (evidence: `readme.md`)
- Data Cleaning → Cleaned Domain CSVs: writes CSVs (evidence: `readme.md`)
- Cleaned Domain CSVs → Spark Transform: reads inputs (evidence: `readme.md`)
- Spark Transform → Star Schema Loader: provides processed data (evidence: `readme.md`)
- Star Schema Loader → SQLite Warehouse: loads schema (evidence: `warehouse/warehouse.py`)
- SQLite Warehouse → SQL KPI Analysis: queries (evidence: `readme.md`)
- SQLite Warehouse → Streamlit Dashboard: reads data (evidence: `dashboard/app.py`)
- SQLite Warehouse → Power BI Export: exports tables (evidence: `pipelines/powerbi_export.py`)
- Power BI Export → Power BI Dashboard: supplies CSVs (evidence: `readme.md`)
- Daily Scheduler → Daily Update Pipeline: launches (evidence: `pipelines/scheduler.py`)
- Daily Update Pipeline → SQLite Warehouse: inserts orders (evidence: `pipelines/daily_pipeline.py`)
- Daily Update Pipeline → Pipeline Run Log: records runs (evidence: `pipelines/daily_pipeline.py`)
- Daily Update Pipeline → Power BI Dashboard: re-exports tables (evidence: `pipelines/daily_pipeline.py`)
- Run History Viewer → Pipeline Run Log: reads history (evidence: `pipelines/view_logs.py`)
