How vsang916/khodl works
This is a data-warehouse and price-mining project for Vietnamese agricultural produce. Its main workflow collects monthly market prices and daily Winmart prices, stages raw and processed data in MinIO, transforms them into ClickHouse star schemas with snapshot-based CDC, then serves OLAP and mining results through Superset dashboards. The README documents the broader capabilities; the sampled source confirms key ingestion, transformation, CDC, and warehouse paths. Some OLAP, ML, and dashboard links are documented rather than verified from their implementation excerpts.
Main entry points: ttns_price_pipeline.py Market DAG · winmart_api_pipeline.py Winmart DAG · publish_ml.py Publish results · olap.py OLAP service
Components 22
Collection
- Market DAG
luong/dags/ttns_price_pipeline.py - Market scraper
luong/dags/pipelines/ttns/scraper.py - Winmart DAG
luong/dags/winmart_api_pipeline.py - API connector
luong/dags/pipelines/winmart/api.py
Staging and CDC
- MinIO lake
- Lake access
luong/dags/pipelines/common/minio_helper.py - Export parser
luong/dags/pipelines/ttns/export.py - Market cleaning
luong/dags/pipelines/ttns/cleaning.py - Retail transform
luong/dags/pipelines/winmart/transform.py - Snapshot CDC
luong/dags/pipelines/common/cdc.py - Market CDC
luong/dags/pipelines/ttns/cdc.py - Retail CDC
luong/dags/pipelines/winmart/cdc.py
Warehouse
- ClickHouse
- Warehouse access
luong/dags/pipelines/common/clickhouse_helper.py - Market star schema
luong/dags/pipelines/ttns/schema.py - Retail schema
luong/dags/pipelines/winmart/schema.py
Analytics
- OLAP service
luong/dags/pipelines/ttns/olap.py - Price mining
luong/ml/KhaiPha_GiaNongSan.py - Publish results
luong/ml/publish_ml.py
Presentation
- Superset dashboards
luong/superset/bootstrap.py
Other components
- Market data
- Winmart API
Connections 27
- Market DAG → Market scraper: runs extraction
luong/dags/ttns_price_pipeline.py - Market scraper → Market data: fetches prices
README.md - Winmart DAG → API connector: runs fetch
luong/dags/winmart_api_pipeline.py - API connector → Winmart API: fetches pages
luong/dags/pipelines/winmart/api.py - Market DAG → MinIO lake: stores raw files
luong/dags/ttns_price_pipeline.py - Winmart DAG → MinIO lake: stores raw files
luong/dags/winmart_api_pipeline.py - Lake access → MinIO lake: reads and writes
luong/dags/pipelines/common/minio_helper.py - Market DAG → Export parser: parses exports
luong/dags/pipelines/ttns/export.py - Market DAG → Market cleaning: cleans records
luong/dags/ttns_price_pipeline.py - Export parser → Market cleaning: normalizes input
luong/dags/pipelines/ttns/export.py - Winmart DAG → Retail transform: builds tables
luong/dags/pipelines/winmart/transform.py - Market DAG → Market CDC: computes changes
luong/dags/ttns_price_pipeline.py - Market CDC → Snapshot CDC: uses snapshot logic
luong/dags/pipelines/ttns/cdc.py - Winmart DAG → Retail CDC: computes changes
luong/dags/winmart_api_pipeline.py - Retail CDC → Snapshot CDC: uses snapshot logic
luong/dags/pipelines/winmart/cdc.py - Market DAG → ClickHouse: loads warehouse
luong/dags/ttns_price_pipeline.py - Winmart DAG → ClickHouse: loads warehouse
luong/dags/winmart_api_pipeline.py - Warehouse access → ClickHouse: queries and loads
luong/dags/pipelines/common/clickhouse_helper.py - Market DAG → Market star schema: creates tables
luong/dags/ttns_price_pipeline.py - Winmart DAG → Retail schema: creates tables
luong/dags/winmart_api_pipeline.py - Market DAG → OLAP service: builds cuboids
luong/dags/ttns_price_pipeline.py - ClickHouse → OLAP service: provides warehouse data
README.md - Price mining → ClickHouse: reads processed data
README.md - Price mining → Publish results: publishes findings
luong/ml/KhaiPha_GiaNongSan.py - Publish results → ClickHouse: writes results
README.md - OLAP service → Superset dashboards: feeds dashboards
README.md - Publish results → Superset dashboards: supplies mining views
README.md