How anesriad/regression_ml_endtoend works
1. Overall shape
- This repo is an end-to-end housing price regression system built around a single ML pipeline, with separate paths for offline training, online inference, batch prediction, and two user-facing services: a FastAPI backend and a Streamlit dashboard.
2. Data preparation boundary
- The core data flow starts in src/feature_pipeline/.
- load.py performs the time-aware split: training data is before 2020, evaluation data is 2020–2021, and holdout data is 2022+.
- preprocess.py cleans and normalizes data before modeling.
- feature_engineering.py creates model inputs such as date-derived features plus categorical encodings.
3. Leakage prevention is a first-class design constraint
- The pipeline avoids random splitting and instead uses time-based splits.
- Encoders are fit only on training data and persisted for later reuse.
- Training/inference schema alignment is enforced so serving uses the same feature set as training.
4. Training and model selection subsystem
- src/training_pipeline/ contains the model lifecycle:
- train.py for baseline XGBoost training
- tune.py for Optuna-driven hyperparameter search
- eval.py for metrics and comparison
- This is the main offline learning boundary where the model is fit and assessed before deployment.
5. Experiment tracking and reproducibility
- MLflow is used to record runs and tuning results.
- Trained artifacts are persisted under models/, including the XGBoost model and saved encoders.
- The repository includes configs/mlflow_config.yml and notebook-driven experimentation history in notebooks/.
6. Inference subsystem
- src/inference_pipeline/inference.py is the production transformation-and-prediction path.
- It loads the saved model plus frequency and target encoders, applies the same feature logic as training, and outputs predictions.
- This subsystem is the boundary between trained artifacts and any serving layer.
7. Batch prediction path
- src/batch/run_monthly.py runs monthly batch scoring, likely over holdout or period-based data.
- This is a separate execution mode from low-latency serving and is useful for scheduled reporting or offline production scoring.
8. API serving layer
- src/api/main.py exposes the model through FastAPI.
- The API is the primary serving boundary for prediction requests and health checks, and it integrates with S3 for model/data access in production.
- The readme indicates the API is containerized and deployed independently from the dashboard.
9. Interactive dashboard layer
- app.py is a Streamlit app that provides an interactive front end.
- It calls the FastAPI service for predictions rather than embedding model logic directly, which keeps the UI separate from inference logic.
- The dashboard also handles filtering and visualization of predictions versus actuals.
10. Cloud and deployment architecture
- The project is designed for AWS:
- S3 for data/model storage
- ECR for container images
- ECS/Fargate for running the API and Streamlit services
- ALB for routing traffic
- housing-api-task-def.json and streamlit-task-def.json are deployment descriptors for those services.
- Dockerfile and Dockerfile.streamlit define the two runtime containers.
11. Quality and validation
- configs/ge_expectations.yml suggests Great Expectations-based data checks.
- tests/ covers feature logic, training, inference, and smoke tests, which supports isolation of pipeline stages and prevents regressions.
12. What is runtime vs. research
- The runtime architecture is the pipeline + API + dashboard + batch job.
- The notebooks are experimentation and analysis artifacts, not part of the deployed serving path, but they explain how the current feature set and model choices were developed.
13. Primary architecture diagram takeaway
- The cleanest diagram is a layered flow:
- raw housing data → feature pipeline → training/tuning/evaluation → persisted model artifacts
- persisted model artifacts + feature pipeline logic → inference/batch scoring
- inference service → API → Streamlit UI
- all of it deployed as containers on AWS with S3-backed storage and MLflow for experiment tracking.
Main entry points: feature_engineering.py Engineer features · train.py Train model · inference.py Predict · tune.py Tune params
Components 19
Data prep
Time-aware feature pipeline and data validation
- Housing data · Incoming housing records before split and cleaning
- Split data · Chronological train/eval/holdout partitioning to avoid leakage
src/feature_pipeline/load.py - Clean data · Normalization and cleaning before feature creation
src/feature_pipeline/preprocess.py - Engineer features · Date-derived features and categorical encodings
src/feature_pipeline/feature_engineering.py - Data checks · Great Expectations rules for data quality and schema consistency
configs/ge_expectations.yml
Training
Offline model fitting, tuning, and evaluation
- Train model · Baseline XGBoost model fitting
src/training_pipeline/train.py - Tune params · Hyperparameter search and best-run selection
src/training_pipeline/tune.py - Evaluate model · Metric calculation and model comparison
src/training_pipeline/eval.py - Track runs · MLflow configuration and run logging
configs/mlflow_config.yml - Model artifact · Saved XGBoost model used for later inference
models/xgb_model.pkl - Encoders · Saved frequency and target encoders reused in serving
models/freq_encoder.pkl
Serving
Inference, API, batch scoring, and UI
- Predict · Loads artifacts, applies training-aligned transforms, and outputs predictions
src/inference_pipeline/inference.py - Monthly batch · Scheduled offline scoring over period-based data
src/batch/run_monthly.py - FastAPI · Prediction and health-check service boundary
src/api/main.py - Streamlit app · Interactive UI that calls the API for predictions
app.py
Deployment
Containers and AWS task definitions
- API image · Container definition for the FastAPI runtime
Dockerfile - UI image · Container definition for the Streamlit runtime
Dockerfile.streamlit - API task · AWS task definition for the API service
housing-api-task-def.json - UI task · AWS task definition for the Streamlit service
streamlit-task-def.json
Connections 21
- Housing data → Split data: split · Time-based partitioning starts from raw housing records
- Split data → Clean data: clean · Split datasets are cleaned before features are built
- Clean data → Engineer features: featurize · Preprocessed rows are transformed into model inputs
- Data checks → Split data: validate · Quality checks constrain the upstream data boundary
- Engineer features → Train model: fit · Training consumes engineered features
- Engineer features → Tune params: search · Hyperparameter tuning reuses the same feature set
- Engineer features → Evaluate model: score · Evaluation compares predictions on held-out data
- Train model → Model artifact: save · Baseline training persists the fitted model
- Train model → Encoders: save · Training persists encoders for later reuse
- Tune params → Model artifact: best model · Best tuned model is written as the deployable artifact
- Train model → Track runs: log · Training runs are recorded for reproducibility
- Tune params → Track runs: log · Tuning trials and best params are tracked
- Model artifact → Predict: load · Inference loads the saved model artifact
- Encoders → Predict: load · Inference loads persisted encoders to match training
- Predict → Monthly batch: score · Monthly batch jobs reuse the same prediction path
- Predict → FastAPI: serve · FastAPI exposes the inference pipeline
- FastAPI → Streamlit app: requests · Dashboard submits prediction requests to the API
- API image → FastAPI: build · API container packages the serving runtime
- UI image → Streamlit app: build · UI container packages the dashboard runtime
- API task → API image: deploy · ECS task runs the API image
- UI task → UI image: deploy · ECS task runs the Streamlit image