# francesshen-lawxtech/tpe-real-estate-ai architecture

> Architecture diagram of the GitHub repository francesshen-lawxtech/tpe-real-estate-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/francesshen-lawxtech/tpe-real-estate-ai
- Repository: https://github.com/francesshen-lawxtech/tpe-real-estate-ai

## Overview

1. **Repository shape**
   - This repo is not a single codebase, but a **family of related Taipei real-estate projects**.
   - The **main production-style application** appears to be under `project/`, supported by a model-training pipeline in `train/`.
   - The other top-level folders (`kevin/`, `frances/`, `永慶房屋/`, `實價登錄最終版/`, `final/`, `taipei-real-estate-quant-engine/`, `taipeihouseapi/`) look like **earlier iterations, parallel experiments, or data-prep pipelines**.

2. **Primary runtime boundary: `project/`**
   - `project/main.py` is the entry point for the main app, described in the README as a **FastAPI** service.
   - `project/api/` contains the business logic split into three API domains:
     - `predict_v3.py` for price prediction
     - `recommender.py` for recommendations
     - `market_trends.py` for trend analysis
   - `project/templates/` and `project/static/` provide the browser-facing UI and map/data assets, so this app serves both **API responses and HTML pages**.

3. **Main user-facing features**
   - The application exposes three logical product areas:
     - **Prediction**: estimate reasonable price ranges
     - **Recommendation**: likely neighborhood/house suggestions
     - **Market trends**: interpret market movement and context
   - The HTML pages in `project/templates/` map directly to these areas: `page_predict.html`, `page_recommend.html`, `page_trends.html`.

4. **Model-serving layer**
   - `project/models/` contains six serialized **LightGBM** models:
     - three for **listing price** ranges
     - three for **actual transaction price** ranges
   - The design uses **quantile-style outputs**: low / mid / high, corresponding to roughly P10 / P50 / P90.
   - These models are the core computational boundary: the API layer loads them and turns feature input into price-range output.

5. **Feature and geography dependencies**
   - The model is not just a plain regression over house attributes; it includes **location and transit context**.
   - Static JSON files under `project/static/` provide:
     - Taipei district boundaries
     - district code mappings
     - MRT station coordinate and cluster mappings
   - This means the prediction path depends on **geospatial preprocessing**: district encoding, MRT proximity, and station clustering.

6. **Training pipeline: `train/`**
   - `train/train_v3.py` is the main training script referenced by the README.
   - Training outputs the model artifacts consumed by `project/`:
     - `lgbm_listing_*.pkl`
     - `lgbm_real_*.pkl`
     - `district_map.json`
     - `mrt_cluster_map.json`
   - `train/train_all.py` and `train/train_predict.py` suggest earlier or auxiliary training variants, but `train_v3.py` is the important one for the current architecture.

7. **Data lineage for training**
   - The README says training uses two core datasets:
     - `house_prices_taipei.csv` for historical transaction prices
     - `house_prediction_data.csv` for current listing prices
   - The feature set combines:
     - physical property attributes
     - address / district
     - coordinates
     - distance to MRT
     - time of transaction
     - property type flags
   - The data flow is: **raw/merged CSVs → cleaned feature table → model training → serialized models and mapping tables**.

8. **ETL and cleaning boundary**
   - Several folders contain scripts and CSVs for scraping, cleaning, and merging data before training.
   - This includes sources such as:
     - real-estate listing sites
     - official transaction records
     - district population data
     - salary/age data
     - MRT station coordinates
   - These inputs are combined into merged datasets used downstream by model training and APIs.

9. **Historical / alternate data pipelines**
   - `kevin/`, `frances/`, and `永慶房屋/` each contain crawler, cleaning, and merge scripts plus district CSVs.
   - `實價登錄最終版/` and `final/` also contain ETL scripts and final merged CSVs.
   - Architecturally, these directories represent **parallel data-prep implementations**, likely created by different contributors or at different stages, rather than one unified pipeline.

10. **Alternative dashboards / API prototypes**
   - `taipeihouseapi/` and `taipei-real-estate-quant-engine/` look like earlier standalone apps or dashboards.
   - They include their own HTML and API entry files, plus district/MRT JSON files.
   - These are best treated as **prototype presentation layers** feeding from similar Taipei housing data, not the main `project/` runtime.

11. **Current deployment shape**
   - The likely deployable stack is:
     - FastAPI backend in `project/main.py`
     - API modules in `project/api/`
     - pretrained LightGBM artifacts in `project/models/`
     - static geographic datasets in `project/static/`
     - HTML templates in `project/templates/`
   - That is the main boundary to diagram if you want the active product path.

12. **What to emphasize in an architecture diagram**
   - Show three layers:
     - **Data acquisition / cleaning**
     - **Model training**
     - **Online serving**
   - Also show the two external data families feeding the system:
     - **real-estate market data**
     - **geospatial / district / MRT context**
   - The main runtime dependency is the **trained model artifacts**, while the rest of the repo mostly supports generating and serving them.

Main entry points: [`final`](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/final) (Final ETL), [`train/train_v3.py`](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/train/train_v3.py) (Train v3), [`project/main.py`](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/project/main.py) (FastAPI app), [`project/api/predict_v3.py`](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/project/api/predict_v3.py) (Predict API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_data["Data pipelines"]
  node_node_real_estate_sources[("Market data<br/>external data")]
  node_node_geo_context[("Geo context<br/>external data")]
  node_node_kevin_pipeline["Kevin pipeline<br/>etl prototype"]
  node_node_frances_pipeline["Frances pipeline<br/>etl prototype"]
  node_node_yongqing_pipeline["Yongqing pipeline<br/>etl prototype"]
  node_node_final_pipeline["Final ETL<br/>etl consolidation"]
  node_node_real_price_pipeline["Real price prep<br/>etl finalizer"]
end

subgraph group_grp_train["Training"]
  node_node_train_v3["Train v3<br/>training script<br/>[train_v3.py]"]
  node_node_train_variants["Train variants<br/>training variants<br/>[train_all.py]"]
  node_node_model_artifacts[("Model artifacts<br/>serialized models")]
  node_node_mapping_artifacts["Mapping tables<br/>feature maps"]
end

subgraph group_grp_serving["Online app"]
  node_node_main_app["FastAPI app<br/>web service<br/>[main.py]"]
  node_node_predict_api["Predict API<br/>inference module<br/>[predict_v3.py]"]
  node_node_recommend_api["Recommend API<br/>ranking module<br/>[recommender.py]"]
  node_node_trends_api["Trends API<br/>analysis module<br/>[market_trends.py]"]
  node_node_templates["HTML pages<br/>presentation"]
  node_node_static_geo["Static geo<br/>static assets"]
  node_node_other_apps["Prototype apps<br/>legacy apps"]
end

node_node_real_estate_sources -->|"source data"| node_node_kevin_pipeline
node_node_real_estate_sources -->|"source data"| node_node_frances_pipeline
node_node_real_estate_sources -->|"source data"| node_node_yongqing_pipeline
node_node_geo_context -->|"geo inputs"| node_node_kevin_pipeline
node_node_geo_context -->|"geo inputs"| node_node_yongqing_pipeline
node_node_kevin_pipeline -->|"merged output"| node_node_final_pipeline
node_node_frances_pipeline -->|"merged output"| node_node_final_pipeline
node_node_yongqing_pipeline -->|"merged output"| node_node_final_pipeline
node_node_final_pipeline -->|"final datasets"| node_node_real_price_pipeline
node_node_final_pipeline -->|"training data"| node_node_train_v3
node_node_real_price_pipeline -->|"training data"| node_node_train_v3
node_node_train_variants -.->|"experiments"| node_node_train_v3
node_node_train_v3 -->|"exports"| node_node_model_artifacts
node_node_train_v3 -->|"exports"| node_node_mapping_artifacts
node_node_model_artifacts -->|"loads models"| node_node_predict_api
node_node_mapping_artifacts -->|"feature maps"| node_node_predict_api
node_node_static_geo -->|"geo lookup"| node_node_predict_api
node_node_predict_api -->|"route"| node_node_main_app
node_node_recommend_api -->|"route"| node_node_main_app
node_node_trends_api -->|"route"| node_node_main_app
node_node_templates -->|"pages"| node_node_main_app
node_node_static_geo -->|"static files"| node_node_main_app
node_node_other_apps -.->|"prototype data"| node_node_static_geo
node_node_other_apps -.->|"legacy flow"| node_node_real_estate_sources

click node_node_kevin_pipeline "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/kevin"
click node_node_frances_pipeline "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/frances"
click node_node_yongqing_pipeline "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/永慶房屋"
click node_node_final_pipeline "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/final"
click node_node_real_price_pipeline "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/實價登錄最終版"
click node_node_train_v3 "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/train/train_v3.py"
click node_node_train_variants "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/train/train_all.py"
click node_node_model_artifacts "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/train/models_v3_mrt_cluster"
click node_node_mapping_artifacts "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/train/models_v3_mrt_cluster"
click node_node_main_app "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/project/main.py"
click node_node_predict_api "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/project/api/predict_v3.py"
click node_node_recommend_api "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/project/api/recommender.py"
click node_node_trends_api "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/project/api/market_trends.py"
click node_node_templates "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/project/templates"
click node_node_static_geo "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/project/static"
click node_node_other_apps "https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/taipei-real-estate-quant-engine"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_real_estate_sources,node_node_geo_context,node_node_kevin_pipeline,node_node_frances_pipeline,node_node_yongqing_pipeline,node_node_final_pipeline,node_node_real_price_pipeline toneBlue
class node_node_train_v3,node_node_train_variants,node_node_model_artifacts,node_node_mapping_artifacts toneAmber
class node_node_main_app,node_node_predict_api,node_node_recommend_api,node_node_trends_api,node_node_templates,node_node_static_geo,node_node_other_apps toneMint
```

## Components

### Data pipelines

Parallel scraping, cleaning, and merge pipelines that prepare Taipei housing datasets.

- **Market data**: external data · Listing and official transaction data plus related source CSVs feeding the ETL pipelines.
- **Geo context**: external data · District, MRT, population, and salary context used as model features.
- [Kevin pipeline](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/kevin): etl prototype · Early crawler/clean/merge implementation with local JSON and CSV conversions. · `kevin`
- [Frances pipeline](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/frances): etl prototype · Parallel scraping and merge flow with per-district listing and age datasets. · `frances`
- [Yongqing pipeline](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/永慶房屋): etl prototype · Crawler and cleaning pipeline for housing output and civic context CSVs. · `永慶房屋`
- [Final ETL](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/final): etl consolidation · Later consolidation scripts that merge real-estate, population, and MRT inputs. · `final`
- [Real price prep](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/實價登錄最終版): etl finalizer · Alternate final-price processing and merge scripts for historical transaction data. · `實價登錄最終版`

### Training

Model-building pipeline that turns prepared data into reusable artifacts.

- [Train v3](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/train/train_v3.py): training script · Main training entrypoint that builds the current LightGBM model set. · `train/train_v3.py`
- [Train variants](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/train/train_all.py): training variants · Earlier or auxiliary training scripts used for experimentation and comparisons. · `train/train_all.py`
- [Model artifacts](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/train/models_v3_mrt_cluster): serialized models · Quantile-style LightGBM models for listing and real transaction price ranges. · `train/models_v3_mrt_cluster`
- [Mapping tables](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/train/models_v3_mrt_cluster): feature maps · District and MRT cluster mapping tables consumed during online inference. · `train/models_v3_mrt_cluster`

### Online app

FastAPI application that serves prediction, recommendation, and trend features.

- [FastAPI app](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/project/main.py): web service · Main runtime entrypoint that wires routes, templates, static assets, and API modules. · `project/main.py`
- [Predict API](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/project/api/predict_v3.py): inference module · Price prediction logic that loads models and produces low/mid/high outputs. · `project/api/predict_v3.py`
- [Recommend API](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/project/api/recommender.py): ranking module · Neighborhood and house recommendation logic. · `project/api/recommender.py`
- [Trends API](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/blob/main/project/api/market_trends.py): analysis module · Market trend interpretation and context features. · `project/api/market_trends.py`
- [HTML pages](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/project/templates): presentation · Browser-facing pages for predict, recommend, trends, and the main index. · `project/templates`
- [Static geo](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/project/static): static assets · District boundaries, MRT stations, and cluster maps used by the UI and inference. · `project/static`
- [Prototype apps](https://github.com/francesshen-lawxtech/tpe-real-estate-ai/tree/main/taipei-real-estate-quant-engine): legacy apps · Earlier standalone dashboard/API prototypes outside the main production path. · `taipei-real-estate-quant-engine`

## Connections

- Market data → Kevin pipeline: source data · Raw market data enters one ETL branch.
- Market data → Frances pipeline: source data · Raw market data enters another ETL branch.
- Market data → Yongqing pipeline: source data · Listing and transaction sources feed the Yongqing branch.
- Geo context → Kevin pipeline: geo inputs · District and MRT context enriches the Kevin pipeline.
- Geo context → Yongqing pipeline: geo inputs · Population, salary, and MRT data enrich the Yongqing pipeline.
- Kevin pipeline → Final ETL: merged output · Cleaned branch output can be consolidated downstream.
- Frances pipeline → Final ETL: merged output · Parallel cleaned output is combined in the final merge path.
- Yongqing pipeline → Final ETL: merged output · Cleaned district-level data flows into final consolidation.
- Final ETL → Real price prep: final datasets · Prepared CSVs feed the alternate real-price processing branch.
- Final ETL → Train v3: training data · Prepared tables become inputs for the main training job.
- Real price prep → Train v3: training data · Historical transaction datasets are used for model fitting.
- Train variants → Train v3: experiments · Earlier training scripts represent alternative or older model-building flows.
- Train v3 → Model artifacts: exports · Training produces LightGBM model files for serving.
- Train v3 → Mapping tables: exports · Training also writes district and MRT cluster maps.
- Model artifacts → Predict API: loads models · Prediction service reads the serialized model set.
- Mapping tables → Predict API: feature maps · Prediction service uses mapping tables for feature encoding.
- Static geo → Predict API: geo lookup · Static district and MRT assets support inference-time location handling.
- Predict API → FastAPI app: route · Prediction logic is exposed through the FastAPI application.
- Recommend API → FastAPI app: route · Recommendation logic is exposed through the FastAPI application.
- Trends API → FastAPI app: route · Trend analysis logic is exposed through the FastAPI application.
- HTML pages → FastAPI app: pages · HTML templates are rendered by the main application.
- Static geo → FastAPI app: static files · The app serves geographic JSON and client-side assets.
- Prototype apps → Static geo: prototype data · Legacy prototype apps carry similar district and MRT assets.
- Prototype apps → Market data: legacy flow · Prototype apps reflect earlier experimental data-processing paths.