# fahminlb33/bogor-house-price architecture

> Architecture diagram of the GitHub repository fahminlb33/bogor-house-price, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fahminlb33/bogor-house-price
- Repository: https://github.com/fahminlb33/bogor-house-price

## Overview

##### Project type and purpose
- This is an end-to-end data and ML pipeline for descriptive and predictive house-price analysis in Bogor. It comprises web scraping, an ELT pipeline (DuckDB + dbt), exploratory notebooks, model training (LightGBM + Optuna + MLflow), a prediction API, a Streamlit front end, and a Power BI dashboard.

##### File structure
- Top-level:
- data/: raw data store
- notebooks/: EDA & model experiments
- scripts/: ingestion (download-*.py), indexing, DuckDB loader, hyperparam tuning
- src/: four subprojects
– scrapy-rumah: Scrapy spiders for rumah123 & OSM
– dbt-rumah: dbt project with staging → intermediate → marts layers
– mcp-rumah: Python API server (embeddings, price_predictor, FastAPI/Flask)
– streamlit-rumah: Streamlit app (UI pages, utils for DB/LLM/price prediction)
- docker-compose.yml + docker/…: orchestrates DB, dbt, API, Streamlit, MLflow server
- Makefile / mise.toml: scripts orchestration

##### README
- Confirms components: Scrapy, DuckDB + dbt ELT, pandas EDA, LightGBM modeling, Optuna + MLflow tuning, Power BI, Streamlit.

- Data Sources: rumah123 website, OSM
- Scraper Service (Scrapy container)
- Raw Data Store (data/ folder, DuckDB file)
- Ingestion & Load Scripts (scripts/download-*.py, scripts/load-duckdb.py)
- ELT Orchestrator (dbt container)
- dbt Layers: Staging → Intermediate → Marts (sql models, macros, seeds)
- Analytical Notebooks (Jupyter, for EDA & feature engineering)
- ML Pipeline:
– Feature engineering code (notebooks/scripts)
– Model training (LightGBM) & hyperparam tuning (Optuna)
– Tracking & model registry (MLflow server)
- Model artifact storage (MLflow artifacts, streamlit assets)
- Prediction API (mcp-rumah service)
- User Interfaces:
– Streamlit app (streamlit-rumah)
– Power BI dashboard (dashboard.pbix)
- Infrastructure: Docker Compose, environment configs

1. Scraper → Raw Data Store
2. Ingestion Scripts → DuckDB (raw table)
3. dbt runs against DuckDB → transformed tables (staging → marts)
4. Notebooks read transformed tables → produce features & training data
5. ML pipeline reads features → trains models → logs to MLflow
6. Model artifacts → served by Prediction API
7. Streamlit UI & Power BI query transformed tables and/or API for predictions
8. Docker Compose ties all containers & networks

c. Architectural patterns / principles:
- ELT with dbt (separation of staging/intermediate/marts)
- Modular “micro-services” for scraper, API, UI
- Experiment tracking decoupled via MLflow
- Infrastructure as code (Docker Compose, Makefile)

d. Key technologies:
- Python / Scrapy, DuckDB, dbt, pandas, LightGBM, Optuna, MLflow, FastAPI/Flask, Streamlit, Power BI, Docker

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Ingestion Layer
  subgraph "Ingestion Layer"
    direction TB
    DS1["Rumah123 Website"]:::ingestion
    DS2["OpenStreetMap (OSM)"]:::ingestion
    Scraper["Scraper Service (Scrapy)"]:::ingestion
    RawStore[(Raw Data Store)]:::ingestion
    Scripts[/Ingestion & Load Scripts/]:::ingestion
  end

  %% Transformation Layer
  subgraph "Transformation Layer"
    direction TB
    ELT["ELT Orchestrator (dbt)"]:::transformation
    subgraph "dbt Layers"
      direction TB
      Staging[(Staging)]:::transformation
      Intermediate[(Intermediate)]:::transformation
      Marts[(Marts)]:::transformation
    end
  end

  %% Modeling Layer
  subgraph "Modeling Layer"
    direction TB
    Notebooks[/Analytical Notebooks/]:::modeling
    TrainFE["Feature Engineering"]:::modeling
    TrainModel["Model Training (LightGBM)"]:::modeling
    Tune["Hyperparam Tuning (Optuna)"]:::modeling
    MLflow["MLflow Server"]:::modeling
    Artifacts[(Model Artifacts)]:::modeling
  end

  %% Serving Layer
  subgraph "Serving Layer"
    direction TB
    API["Prediction API (/predict)"]:::serving
  end

  %% Presentation Layer
  subgraph "Presentation Layer"
    direction TB
    Streamlit["Streamlit App"]:::ui
    PowerBI["Power BI Dashboard"]:::ui
  end

  %% Infrastructure
  subgraph "Infrastructure & Orchestration"
    direction TB
    DC["Docker Compose"]:::infra
    Env["docker/db.env.example"]:::infra
    Makefile["Makefile / mise.toml"]:::infra
  end

  %% Data Flows
  DS1 --> Scraper
  DS2 --> Scraper
  Scraper --> RawStore
  Scripts --> RawStore
  Scripts --> ELT
  ELT --> Staging
  Staging --> Intermediate
  Intermediate --> Marts
  Marts --> Notebooks
  Notebooks --> TrainFE
  TrainFE --> TrainModel
  TrainModel --> Tune
  Tune --> MLflow
  MLflow --> Artifacts
  Artifacts --> API
  Marts --> Streamlit
  API --> Streamlit
  API --> PowerBI
  DC --> Scraper
  DC --> ELT
  DC --> MLflow
  DC --> API
  DC --> Streamlit

  %% Click Events
  click Scraper "https://github.com/fahminlb33/bogor-house-price/tree/master/src/scrapy-rumah"
  click RawStore "https://github.com/fahminlb33/bogor-house-price/tree/master/data/"
  click Scripts "https://github.com/fahminlb33/bogor-house-price/blob/master/scripts/download-images.py"
  click Scripts "https://github.com/fahminlb33/bogor-house-price/blob/master/scripts/download-osm.py"
  click Scripts "https://github.com/fahminlb33/bogor-house-price/blob/master/scripts/index-data.py"
  click Scripts "https://github.com/fahminlb33/bogor-house-price/blob/master/scripts/load-duckdb.py"
  click ELT "https://github.com/fahminlb33/bogor-house-price/blob/master/src/dbt-rumah/dbt_project.yml"
  click ELT "https://github.com/fahminlb33/bogor-house-price/tree/master/src/dbt-rumah/models/"
  click Staging "https://github.com/fahminlb33/bogor-house-price/tree/master/src/dbt-rumah/models/staging/"
  click Intermediate "https://github.com/fahminlb33/bogor-house-price/tree/master/src/dbt-rumah/models/intermediate/"
  click Marts "https://github.com/fahminlb33/bogor-house-price/tree/master/src/dbt-rumah/models/marts/"
  click Notebooks "https://github.com/fahminlb33/bogor-house-price/tree/master/notebooks/"
  click TrainModel "https://github.com/fahminlb33/bogor-house-price/blob/master/notebooks/train-eval.ipynb"
  click TrainFE "https://github.com/fahminlb33/bogor-house-price/blob/master/notebooks/train-feature-engineering.ipynb"
  click Tune "https://github.com/fahminlb33/bogor-house-price/blob/master/scripts/tune.py"
  click MLflow "https://github.com/fahminlb33/bogor-house-price/blob/master/docker-compose.yml"
  click API "https://github.com/fahminlb33/bogor-house-price/blob/master/src/mcp-rumah/server.py"
  click Streamlit "https://github.com/fahminlb33/bogor-house-price/tree/master/src/streamlit-rumah/"
  click PowerBI "https://github.com/fahminlb33/bogor-house-price/blob/master/src/dashboard.pbix"
  click Env "https://github.com/fahminlb33/bogor-house-price/blob/master/docker/db.env.example"
  click Makefile "https://github.com/fahminlb33/bogor-house-price/tree/master/Makefile"
  click Makefile "https://github.com/fahminlb33/bogor-house-price/blob/master/mise.toml"

  %% Styles
  classDef ingestion fill:#D0E6FF,stroke:#0366d6,stroke-width:1px
  classDef transformation fill:#DFFFE0,stroke:#3b8f00,stroke-width:1px
  classDef modeling fill:#FFE5B4,stroke:#d87a00,stroke-width:1px
  classDef serving fill:#E5CCFF,stroke:#7a2ed8,stroke-width:1px
  classDef ui fill:#E0FFFF,stroke:#008b8b,stroke-width:1px
  classDef infra fill:#F0F0F0,stroke:#666666,stroke-width:1px
```
