How roydontay/sg-property-price-prediction works
Main Components
• Data Acquisition Component:
• Represented by the API_data_extraction.py script. This script is responsible for polling the URA REST API and gathering property transaction data.
• Data Storage & Management:
• The Data directory, which stores the raw data (including archive “Raw Data.zip”) and collated datasets (e.g., Data/Collated_Data_170622_modified). This component holds the data files that move through the pipeline.
• Data Processing, Feature Extraction & EDA:
• Represented by Exploratory_Data_Analysis.ipynb. This component encompasses data cleaning, feature extraction (final_features.pkl), and visual exploratory analysis.
• Model Training & Experimentation:
• Represented by Model_selection_experimentation.ipynb. This notebook shows the experiments with different modeling techniques and selects the best-performing model (Random Forest in this case).
• Model Artifacts:
• The best_model.pkl file holds the serialized version of the final model, and the final_features.pkl stores the features used. Think of these as the endpoints from the training pipeline that will be used in production.
• Model Serving / Frontend Application:
• Represented by app.py, which is a Streamlit application. This application uses the trained model to present interactive predictions to the user.
• That data moves from storage into the Data Processing/EDA phase for cleaning, transformation, and feature extraction.
• Finally, illustrate that these stored model artifacts are loaded by the Streamlit app (app.py) which serves predictions to end users.
Key Architectural Patterns & Technologies
• Since this is an end-to-end pipeline, emphasize the separation of concerns:
• Data Ingestion, Processing, Modeling, and Serving are distinct components.
• Use of scripting and Jupyter notebooks indicates an iterative and exploratory development style common in data science projects.
• The integration of a Streamlit web app means that the model serving component is built using lightweight, interactive Python dashboards.
• The serialized model and feature artifacts (pkl files) act as interfaces between the training and serving stages.
Important Details to Capture
• That initial stages (API extraction and data cleaning) are critical for preparing a robust dataset for modeling.
• The model training component is modular enough to potentially allow swapping or updating the machine learning algorithms.
• The web-based app is the final layer that interacts with the user, making predictions based on input features.
• Optionally, include a note about potential future components (like integrating data from OneMap API) as mentioned in the README for extensibility.