# vsang916/khodl architecture

> Architecture diagram of the GitHub repository vsang916/khodl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vsang916/khodl
- Repository: https://github.com/vsang916/khodl

## Overview

This is a data-warehouse and price-mining project for Vietnamese agricultural produce. Its main workflow collects monthly market prices and daily Winmart prices, stages raw and processed data in MinIO, transforms them into ClickHouse star schemas with snapshot-based CDC, then serves OLAP and mining results through Superset dashboards. The README documents the broader capabilities; the sampled source confirms key ingestion, transformation, CDC, and warehouse paths. Some OLAP, ML, and dashboard links are documented rather than verified from their implementation excerpts.

Main entry points: [`luong/dags/ttns_price_pipeline.py`](https://github.com/vsang916/khodl/blob/main/luong/dags/ttns_price_pipeline.py) (Market DAG), [`luong/dags/winmart_api_pipeline.py`](https://github.com/vsang916/khodl/blob/main/luong/dags/winmart_api_pipeline.py) (Winmart DAG), [`luong/ml/publish_ml.py`](https://github.com/vsang916/khodl/blob/main/luong/ml/publish_ml.py) (Publish results), [`luong/dags/pipelines/ttns/olap.py`](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/olap.py) (OLAP service)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Collection"]
  node_ttns_dag["Market DAG"]
  node_market_scraper["Market scraper<br/>[scraper.py]"]
  node_winmart_dag["Winmart DAG"]
  node_winmart_connector["API connector<br/>[api.py]"]
end

subgraph group_staging["Staging and CDC"]
  node_minio[("MinIO lake")]
  node_minio_helpers["Lake access<br/>[minio_helper.py]"]
  node_ttns_export["Export parser<br/>[export.py]"]
  node_ttns_clean["Market cleaning<br/>[cleaning.py]"]
  node_winmart_transform["Retail transform<br/>[transform.py]"]
  node_cdc_core["Snapshot CDC<br/>[cdc.py]"]
  node_ttns_cdc["Market CDC<br/>[cdc.py]"]
  node_winmart_cdc["Retail CDC<br/>[cdc.py]"]
end

subgraph group_warehouse["Warehouse"]
  node_clickhouse[("ClickHouse")]
  node_ch_helpers["Warehouse access"]
  node_ttns_schema["Market star schema<br/>[schema.py]"]
  node_winmart_schema["Retail schema<br/>[schema.py]"]
end

subgraph group_analysis["Analytics"]
  node_olap["OLAP service<br/>[olap.py]"]
  node_mining["Price mining"]
  node_publish_ml["Publish results<br/>[publish_ml.py]"]
end

subgraph group_serving["Presentation"]
  node_superset["Superset dashboards<br/>[bootstrap.py]"]
end

node_market_source(("Market data"))
node_winmart_source(("Winmart API"))

node_ttns_dag -->|"runs extraction"| node_market_scraper
node_market_scraper -->|"fetches prices"| node_market_source
node_winmart_dag -->|"runs fetch"| node_winmart_connector
node_winmart_connector -->|"fetches pages"| node_winmart_source
node_ttns_dag -->|"stores raw files"| node_minio
node_winmart_dag -->|"stores raw files"| node_minio
node_minio_helpers -->|"reads and writes"| node_minio
node_ttns_dag -->|"parses exports"| node_ttns_export
node_ttns_dag -->|"cleans records"| node_ttns_clean
node_ttns_export -->|"normalizes input"| node_ttns_clean
node_winmart_dag -->|"builds tables"| node_winmart_transform
node_ttns_dag -->|"computes changes"| node_ttns_cdc
node_ttns_cdc -->|"uses snapshot logic"| node_cdc_core
node_winmart_dag -->|"computes changes"| node_winmart_cdc
node_winmart_cdc -->|"uses snapshot logic"| node_cdc_core
node_ttns_dag -->|"loads warehouse"| node_clickhouse
node_winmart_dag -->|"loads warehouse"| node_clickhouse
node_ch_helpers -->|"queries and loads"| node_clickhouse
node_ttns_dag -->|"creates tables"| node_ttns_schema
node_winmart_dag -->|"creates tables"| node_winmart_schema
node_ttns_dag -->|"builds cuboids"| node_olap
node_clickhouse -->|"provides warehouse data"| node_olap
node_mining -->|"reads processed data"| node_clickhouse
node_mining -->|"publishes findings"| node_publish_ml
node_publish_ml -->|"writes results"| node_clickhouse
node_olap -->|"feeds dashboards"| node_superset
node_publish_ml -->|"supplies mining views"| node_superset

click node_ttns_dag "https://github.com/vsang916/khodl/blob/main/luong/dags/ttns_price_pipeline.py"
click node_market_scraper "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/scraper.py"
click node_winmart_dag "https://github.com/vsang916/khodl/blob/main/luong/dags/winmart_api_pipeline.py"
click node_winmart_connector "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/winmart/api.py"
click node_minio_helpers "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/common/minio_helper.py"
click node_ttns_export "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/export.py"
click node_ttns_clean "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/cleaning.py"
click node_winmart_transform "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/winmart/transform.py"
click node_cdc_core "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/common/cdc.py"
click node_ttns_cdc "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/cdc.py"
click node_winmart_cdc "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/winmart/cdc.py"
click node_ch_helpers "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/common/clickhouse_helper.py"
click node_ttns_schema "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/schema.py"
click node_winmart_schema "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/winmart/schema.py"
click node_olap "https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/olap.py"
click node_mining "https://github.com/vsang916/khodl/blob/main/luong/ml/KhaiPha_GiaNongSan.py"
click node_publish_ml "https://github.com/vsang916/khodl/blob/main/luong/ml/publish_ml.py"
click node_superset "https://github.com/vsang916/khodl/blob/main/luong/superset/bootstrap.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ttns_dag,node_market_scraper,node_winmart_dag,node_winmart_connector toneBlue
class node_minio,node_minio_helpers,node_ttns_export,node_ttns_clean,node_winmart_transform,node_cdc_core,node_ttns_cdc,node_winmart_cdc toneAmber
class node_clickhouse,node_ch_helpers,node_ttns_schema,node_winmart_schema,node_winmart_source toneMint
class node_olap,node_mining,node_publish_ml toneRose
class node_superset,node_market_source toneIndigo
```

## Components

### Collection

- [Market DAG](https://github.com/vsang916/khodl/blob/main/luong/dags/ttns_price_pipeline.py): `luong/dags/ttns_price_pipeline.py`
- [Market scraper](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/scraper.py): `luong/dags/pipelines/ttns/scraper.py`
- [Winmart DAG](https://github.com/vsang916/khodl/blob/main/luong/dags/winmart_api_pipeline.py): `luong/dags/winmart_api_pipeline.py`
- [API connector](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/winmart/api.py): `luong/dags/pipelines/winmart/api.py`

### Staging and CDC

- **MinIO lake**
- [Lake access](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/common/minio_helper.py): `luong/dags/pipelines/common/minio_helper.py`
- [Export parser](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/export.py): `luong/dags/pipelines/ttns/export.py`
- [Market cleaning](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/cleaning.py): `luong/dags/pipelines/ttns/cleaning.py`
- [Retail transform](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/winmart/transform.py): `luong/dags/pipelines/winmart/transform.py`
- [Snapshot CDC](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/common/cdc.py): `luong/dags/pipelines/common/cdc.py`
- [Market CDC](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/cdc.py): `luong/dags/pipelines/ttns/cdc.py`
- [Retail CDC](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/winmart/cdc.py): `luong/dags/pipelines/winmart/cdc.py`

### Warehouse

- **ClickHouse**
- [Warehouse access](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/common/clickhouse_helper.py): `luong/dags/pipelines/common/clickhouse_helper.py`
- [Market star schema](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/schema.py): `luong/dags/pipelines/ttns/schema.py`
- [Retail schema](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/winmart/schema.py): `luong/dags/pipelines/winmart/schema.py`

### Analytics

- [OLAP service](https://github.com/vsang916/khodl/blob/main/luong/dags/pipelines/ttns/olap.py): `luong/dags/pipelines/ttns/olap.py`
- [Price mining](https://github.com/vsang916/khodl/blob/main/luong/ml/KhaiPha_GiaNongSan.py): `luong/ml/KhaiPha_GiaNongSan.py`
- [Publish results](https://github.com/vsang916/khodl/blob/main/luong/ml/publish_ml.py): `luong/ml/publish_ml.py`

### Presentation

- [Superset dashboards](https://github.com/vsang916/khodl/blob/main/luong/superset/bootstrap.py): `luong/superset/bootstrap.py`

### Other components

- **Market data**
- **Winmart API**

## Connections

- Market DAG → Market scraper: runs extraction (evidence: `luong/dags/ttns_price_pipeline.py`)
- Market scraper → Market data: fetches prices (evidence: `README.md`)
- Winmart DAG → API connector: runs fetch (evidence: `luong/dags/winmart_api_pipeline.py`)
- API connector → Winmart API: fetches pages (evidence: `luong/dags/pipelines/winmart/api.py`)
- Market DAG → MinIO lake: stores raw files (evidence: `luong/dags/ttns_price_pipeline.py`)
- Winmart DAG → MinIO lake: stores raw files (evidence: `luong/dags/winmart_api_pipeline.py`)
- Lake access → MinIO lake: reads and writes (evidence: `luong/dags/pipelines/common/minio_helper.py`)
- Market DAG → Export parser: parses exports (evidence: `luong/dags/pipelines/ttns/export.py`)
- Market DAG → Market cleaning: cleans records (evidence: `luong/dags/ttns_price_pipeline.py`)
- Export parser → Market cleaning: normalizes input (evidence: `luong/dags/pipelines/ttns/export.py`)
- Winmart DAG → Retail transform: builds tables (evidence: `luong/dags/pipelines/winmart/transform.py`)
- Market DAG → Market CDC: computes changes (evidence: `luong/dags/ttns_price_pipeline.py`)
- Market CDC → Snapshot CDC: uses snapshot logic (evidence: `luong/dags/pipelines/ttns/cdc.py`)
- Winmart DAG → Retail CDC: computes changes (evidence: `luong/dags/winmart_api_pipeline.py`)
- Retail CDC → Snapshot CDC: uses snapshot logic (evidence: `luong/dags/pipelines/winmart/cdc.py`)
- Market DAG → ClickHouse: loads warehouse (evidence: `luong/dags/ttns_price_pipeline.py`)
- Winmart DAG → ClickHouse: loads warehouse (evidence: `luong/dags/winmart_api_pipeline.py`)
- Warehouse access → ClickHouse: queries and loads (evidence: `luong/dags/pipelines/common/clickhouse_helper.py`)
- Market DAG → Market star schema: creates tables (evidence: `luong/dags/ttns_price_pipeline.py`)
- Winmart DAG → Retail schema: creates tables (evidence: `luong/dags/winmart_api_pipeline.py`)
- Market DAG → OLAP service: builds cuboids (evidence: `luong/dags/ttns_price_pipeline.py`)
- ClickHouse → OLAP service: provides warehouse data (evidence: `README.md`)
- Price mining → ClickHouse: reads processed data (evidence: `README.md`)
- Price mining → Publish results: publishes findings (evidence: `luong/ml/KhaiPha_GiaNongSan.py`)
- Publish results → ClickHouse: writes results (evidence: `README.md`)
- OLAP service → Superset dashboards: feeds dashboards (evidence: `README.md`)
- Publish results → Superset dashboards: supplies mining views (evidence: `README.md`)
