How nolanrobbins/housecanary-ml-project works
Main Components
• Data Ingestion & Storage:
• Raw housing data (e.g., "Housing Data.xlsx" and the housing data files inside the data/ directory)
• Additional data files like the housing data dictionary and processed Excel sheets (e.g., "X_train_scaled.xlsx", "X_train_selected.xlsx", "df_cleaned.xlsx")
• Data Preprocessing:
• Jupyter Notebook “Cleaning the Data.ipynb” for data cleaning and preprocessing
• Model Building & Serialization:
• “Model Building Notebook” that processes the cleaned data, conducts ML training, and then serializes outputs
• Persisted artifacts in the data/ folder such as "HouseCanary_model.pkl" (the trained ML model) and "robust_scaler.pkl" (for feature scaling)
• User Interface (Frontend):
• “st_house.py” indicating a Streamlit application for user interaction, where end users input home characteristics
• External Dependencies & Environment:
• “requirements.txt” listing all necessary libraries and dependencies to recreate the environment
Component Relationships and Interactions
• Data flows from raw data sources to preprocessing notebooks:
• Raw data (Excel) is ingested to be cleaned. The cleaning project creates processed datasets (df_cleaned.xlsx) which are then fed into the model-building process.
• The Model Training process:
• The “Model Building Notebook” accesses the cleaned data to train an ML model. It outputs model artifacts (e.g., a serialized .pkl file) that the application will load at runtime.
• User Interaction & Inference:
• The Streamlit app ("st_house.py") acts as the frontend where user inputs are gathered.
• When a user submits home characteristics, the app loads the pre-trained model (and necessary scalers) to compute and return a price estimate.
• Integration with a Sales Funnel:
Architectural Patterns and Design Principles
• Separation of Concerns:
• Clearly segregate data ingestion/preprocessing, model training, and serving/user interaction components.
• Serialization for Portability:
• Use of pickle files to store ML model and scaler indicates modularity, allowing the inference component to remain loosely coupled with the training process.
• Deployment Readiness:
• The inclusion of “requirements.txt” emphasizes environmental reproducibility and deployment considerations.
Final Checklist
• Component (data sources, notebooks, ML model artifact, web interface) has a clear label.
Raw Data → Data Cleaning → Model Building → ML Artifacts → Streamlit (User Inference).
• A note for external API/integration points (the sales funnel filtering system as described in the README).