How fahminlb33/bogor-house-price works
Project type and purpose
• This is an end-to-end data and ML pipeline for descriptive and predictive house-price analysis in Bogor. It comprises web scraping, an ELT pipeline (DuckDB + dbt), exploratory notebooks, model training (LightGBM + Optuna + MLflow), a prediction API, a Streamlit front end, and a Power BI dashboard.
File structure
• Top-level:
• data/: raw data store
• notebooks/: EDA & model experiments
• scripts/: ingestion (download-*.py), indexing, DuckDB loader, hyperparam tuning
• src/: four subprojects
– scrapy-rumah: Scrapy spiders for rumah123 & OSM
– dbt-rumah: dbt project with staging → intermediate → marts layers
– mcp-rumah: Python API server (embeddings, price_predictor, FastAPI/Flask)
– streamlit-rumah: Streamlit app (UI pages, utils for DB/LLM/price prediction)
• docker-compose.yml + docker/…: orchestrates DB, dbt, API, Streamlit, MLflow server
• Makefile / mise.toml: scripts orchestration
README
• Confirms components: Scrapy, DuckDB + dbt ELT, pandas EDA, LightGBM modeling, Optuna + MLflow tuning, Power BI, Streamlit.
• Data Sources: rumah123 website, OSM
• Scraper Service (Scrapy container)
• Raw Data Store (data/ folder, DuckDB file)
• Ingestion & Load Scripts (scripts/download-*.py, scripts/load-duckdb.py)
• ELT Orchestrator (dbt container)
• dbt Layers: Staging → Intermediate → Marts (sql models, macros, seeds)
• Analytical Notebooks (Jupyter, for EDA & feature engineering)
• ML Pipeline:
– Feature engineering code (notebooks/scripts)
– Model training (LightGBM) & hyperparam tuning (Optuna)
– Tracking & model registry (MLflow server)
• Model artifact storage (MLflow artifacts, streamlit assets)
• Prediction API (mcp-rumah service)
• User Interfaces:
– Streamlit app (streamlit-rumah)
– Power BI dashboard (dashboard.pbix)
• Infrastructure: Docker Compose, environment configs
1. Scraper → Raw Data Store
2. Ingestion Scripts → DuckDB (raw table)
3. dbt runs against DuckDB → transformed tables (staging → marts)
4. Notebooks read transformed tables → produce features & training data
5. ML pipeline reads features → trains models → logs to MLflow
6. Model artifacts → served by Prediction API
7. Streamlit UI & Power BI query transformed tables and/or API for predictions
8. Docker Compose ties all containers & networks
c. Architectural patterns / principles:
• ELT with dbt (separation of staging/intermediate/marts)
• Modular “micro-services” for scraper, API, UI
• Experiment tracking decoupled via MLflow
• Infrastructure as code (Docker Compose, Makefile)
d. Key technologies:
• Python / Scrapy, DuckDB, dbt, pandas, LightGBM, Optuna, MLflow, FastAPI/Flask, Streamlit, Power BI, Docker