How kientrung2005/retail-price-intelligence works
Retail Price Intelligence is an orchestrated ETL and analytics platform. Airflow enqueues province/category/store work, workers crawl retailer sites, and raw JSON lands in MinIO. Spark flattens and partitions those records, loading PostgreSQL raw data and processed Parquet. dbt then builds the snowflake-style analytical warehouse, which Power BI uses for regional, retailer, availability, discount, review, and price comparisons. The graph reflects documented orchestration and sampled implementations; several retailer workers and dbt models are represented at their exact enclosing paths because their full source was not sampled.
Components 19
Orchestration
- Airflow DAG
dags/retail_pipeline_dag.py - Task Enqueuer
src/redis/enqueue_tasks.py - Redis Queues
src/redis/client.py
Data Acquisition
- Crawler Runner
src/crawler/run_crawlers.py - CellphoneS Crawler
src/crawler/cellphones_crawler.py - FPT Shop Crawler
src/crawler/fptshop_crawler.py - Dien May Xanh Crawler
src/crawler/dienmayxanh_crawler.py - Review Enrichment
src/utils/review_utils.py
Processing
- MinIO Data Lake
src/storage/minio_client.py - Spark ETL
src/spark/load_to_postgres.py - Spark Connector
src/spark/spark_connect.py - Processed Parquet
src/spark/load_to_postgres.py
Warehouse
- PostgreSQL Raw Data
src/database/create_tables.py - dbt Warehouse Models
dbt/models - Analytical Warehouse
dbt/models/marts
Analytics
- Power BI Dashboard
dashboard/retail-intelligence.pbix - Comparison Results
results
Other components
- Business Analyst
- Retailer Websites
Connections 26
- Business Analyst → Airflow DAG: triggers
- Airflow DAG → Task Enqueuer: runs
- Task Enqueuer → Redis Queues: pushes tasks
- Airflow DAG → Crawler Runner: runs
- Redis Queues → Crawler Runner: dispatches tasks
- Crawler Runner → CellphoneS Crawler: invokes
- Crawler Runner → FPT Shop Crawler: invokes
- Crawler Runner → Dien May Xanh Crawler: invokes
- CellphoneS Crawler → Retailer Websites: fetches listings
- FPT Shop Crawler → Retailer Websites: fetches listings
- Dien May Xanh Crawler → Retailer Websites: fetches listings
- CellphoneS Crawler → Review Enrichment: enriches reviews
- Dien May Xanh Crawler → Review Enrichment: enriches reviews
- Review Enrichment → Redis Queues: caches reviews
- Crawler Runner → MinIO Data Lake: writes raw JSON
- Airflow DAG → Spark ETL: runs
- Spark ETL → Spark Connector: creates session
- Spark ETL → MinIO Data Lake: reads raw JSON
- Spark ETL → Processed Parquet: writes partitions
- Spark ETL → PostgreSQL Raw Data: loads records
- Airflow DAG → dbt Warehouse Models: runs dbt
- dbt Warehouse Models → PostgreSQL Raw Data: reads raw data
- dbt Warehouse Models → Analytical Warehouse: builds marts
- Power BI Dashboard → Analytical Warehouse: reads models
- Business Analyst → Power BI Dashboard: views comparisons
- Power BI Dashboard → Comparison Results: exports visuals