How francesshen-lawxtech/tpe-real-estate-ai works
1. Repository shape
- This repo is not a single codebase, but a family of related Taipei real-estate projects.
- The main production-style application appears to be under project/, supported by a model-training pipeline in train/.
- The other top-level folders (kevin/, frances/, 永慶房屋/, 實價登錄最終版/, final/, taipei-real-estate-quant-engine/, taipeihouseapi/) look like earlier iterations, parallel experiments, or data-prep pipelines.
2. Primary runtime boundary: project/
- project/main.py is the entry point for the main app, described in the README as a FastAPI service.
- project/api/ contains the business logic split into three API domains:
- predict_v3.py for price prediction
- recommender.py for recommendations
- market_trends.py for trend analysis
- project/templates/ and project/static/ provide the browser-facing UI and map/data assets, so this app serves both API responses and HTML pages.
3. Main user-facing features
- The application exposes three logical product areas:
- Prediction: estimate reasonable price ranges
- Recommendation: likely neighborhood/house suggestions
- Market trends: interpret market movement and context
- The HTML pages in project/templates/ map directly to these areas: page_predict.html, page_recommend.html, page_trends.html.
4. Model-serving layer
- project/models/ contains six serialized LightGBM models:
- three for listing price ranges
- three for actual transaction price ranges
- The design uses quantile-style outputs: low / mid / high, corresponding to roughly P10 / P50 / P90.
- These models are the core computational boundary: the API layer loads them and turns feature input into price-range output.
5. Feature and geography dependencies
- The model is not just a plain regression over house attributes; it includes location and transit context.
- Static JSON files under project/static/ provide:
- Taipei district boundaries
- district code mappings
- MRT station coordinate and cluster mappings
- This means the prediction path depends on geospatial preprocessing: district encoding, MRT proximity, and station clustering.
6. Training pipeline: train/
- train/train_v3.py is the main training script referenced by the README.
- Training outputs the model artifacts consumed by project/:
- lgbm_listing_*.pkl
- lgbm_real_*.pkl
- district_map.json
- mrt_cluster_map.json
- train/train_all.py and train/train_predict.py suggest earlier or auxiliary training variants, but train_v3.py is the important one for the current architecture.
7. Data lineage for training
- The README says training uses two core datasets:
- house_prices_taipei.csv for historical transaction prices
- house_prediction_data.csv for current listing prices
- The feature set combines:
- physical property attributes
- address / district
- coordinates
- distance to MRT
- time of transaction
- property type flags
- The data flow is: raw/merged CSVs → cleaned feature table → model training → serialized models and mapping tables.
8. ETL and cleaning boundary
- Several folders contain scripts and CSVs for scraping, cleaning, and merging data before training.
- This includes sources such as:
- real-estate listing sites
- official transaction records
- district population data
- salary/age data
- MRT station coordinates
- These inputs are combined into merged datasets used downstream by model training and APIs.
9. Historical / alternate data pipelines
- kevin/, frances/, and 永慶房屋/ each contain crawler, cleaning, and merge scripts plus district CSVs.
- 實價登錄最終版/ and final/ also contain ETL scripts and final merged CSVs.
- Architecturally, these directories represent parallel data-prep implementations, likely created by different contributors or at different stages, rather than one unified pipeline.
10. Alternative dashboards / API prototypes
- taipeihouseapi/ and taipei-real-estate-quant-engine/ look like earlier standalone apps or dashboards.
- They include their own HTML and API entry files, plus district/MRT JSON files.
- These are best treated as prototype presentation layers feeding from similar Taipei housing data, not the main project/ runtime.
11. Current deployment shape
- The likely deployable stack is:
- FastAPI backend in project/main.py
- API modules in project/api/
- pretrained LightGBM artifacts in project/models/
- static geographic datasets in project/static/
- HTML templates in project/templates/
- That is the main boundary to diagram if you want the active product path.
12. What to emphasize in an architecture diagram
- Show three layers:
- Data acquisition / cleaning
- Model training
- Online serving
- Also show the two external data families feeding the system:
- real-estate market data
- geospatial / district / MRT context
- The main runtime dependency is the trained model artifacts, while the rest of the repo mostly supports generating and serving them.
Main entry points: final Final ETL · train_v3.py Train v3 · main.py FastAPI app · predict_v3.py Predict API
Components 18
Data pipelines
Parallel scraping, cleaning, and merge pipelines that prepare Taipei housing datasets.
- Market data · Listing and official transaction data plus related source CSVs feeding the ETL pipelines.
- Geo context · District, MRT, population, and salary context used as model features.
- Kevin pipeline · Early crawler/clean/merge implementation with local JSON and CSV conversions.
kevin - Frances pipeline · Parallel scraping and merge flow with per-district listing and age datasets.
frances - Yongqing pipeline · Crawler and cleaning pipeline for housing output and civic context CSVs.
永慶房屋 - Final ETL · Later consolidation scripts that merge real-estate, population, and MRT inputs.
final - Real price prep · Alternate final-price processing and merge scripts for historical transaction data.
實價登錄最終版
Training
Model-building pipeline that turns prepared data into reusable artifacts.
- Train v3 · Main training entrypoint that builds the current LightGBM model set.
train/train_v3.py - Train variants · Earlier or auxiliary training scripts used for experimentation and comparisons.
train/train_all.py - Model artifacts · Quantile-style LightGBM models for listing and real transaction price ranges.
train/models_v3_mrt_cluster - Mapping tables · District and MRT cluster mapping tables consumed during online inference.
train/models_v3_mrt_cluster
Online app
FastAPI application that serves prediction, recommendation, and trend features.
- FastAPI app · Main runtime entrypoint that wires routes, templates, static assets, and API modules.
project/main.py - Predict API · Price prediction logic that loads models and produces low/mid/high outputs.
project/api/predict_v3.py - Recommend API · Neighborhood and house recommendation logic.
project/api/recommender.py - Trends API · Market trend interpretation and context features.
project/api/market_trends.py - HTML pages · Browser-facing pages for predict, recommend, trends, and the main index.
project/templates - Static geo · District boundaries, MRT stations, and cluster maps used by the UI and inference.
project/static - Prototype apps · Earlier standalone dashboard/API prototypes outside the main production path.
taipei-real-estate-quant-engine
Connections 24
- Market data → Kevin pipeline: source data · Raw market data enters one ETL branch.
- Market data → Frances pipeline: source data · Raw market data enters another ETL branch.
- Market data → Yongqing pipeline: source data · Listing and transaction sources feed the Yongqing branch.
- Geo context → Kevin pipeline: geo inputs · District and MRT context enriches the Kevin pipeline.
- Geo context → Yongqing pipeline: geo inputs · Population, salary, and MRT data enrich the Yongqing pipeline.
- Kevin pipeline → Final ETL: merged output · Cleaned branch output can be consolidated downstream.
- Frances pipeline → Final ETL: merged output · Parallel cleaned output is combined in the final merge path.
- Yongqing pipeline → Final ETL: merged output · Cleaned district-level data flows into final consolidation.
- Final ETL → Real price prep: final datasets · Prepared CSVs feed the alternate real-price processing branch.
- Final ETL → Train v3: training data · Prepared tables become inputs for the main training job.
- Real price prep → Train v3: training data · Historical transaction datasets are used for model fitting.
- Train variants → Train v3: experiments · Earlier training scripts represent alternative or older model-building flows.
- Train v3 → Model artifacts: exports · Training produces LightGBM model files for serving.
- Train v3 → Mapping tables: exports · Training also writes district and MRT cluster maps.
- Model artifacts → Predict API: loads models · Prediction service reads the serialized model set.
- Mapping tables → Predict API: feature maps · Prediction service uses mapping tables for feature encoding.
- Static geo → Predict API: geo lookup · Static district and MRT assets support inference-time location handling.
- Predict API → FastAPI app: route · Prediction logic is exposed through the FastAPI application.
- Recommend API → FastAPI app: route · Recommendation logic is exposed through the FastAPI application.
- Trends API → FastAPI app: route · Trend analysis logic is exposed through the FastAPI application.
- HTML pages → FastAPI app: pages · HTML templates are rendered by the main application.
- Static geo → FastAPI app: static files · The app serves geographic JSON and client-side assets.
- Prototype apps → Static geo: prototype data · Legacy prototype apps carry similar district and MRT assets.
- Prototype apps → Market data: legacy flow · Prototype apps reflect earlier experimental data-processing paths.