How alaindelong2k/end-to-end-machine-learning-project works
Project type and purpose
• This is an end-to-end machine learning web application (“Student Math Score Predictor”) built in Python. It bundles an ML model training/serving pipeline (data ingestion, transformation, model training, prediction) together with a Streamlit-based user interface for real-time score prediction.
File structure
• Top-level entries:
• app.py – Streamlit application entry point (frontend + serving logic)
• artifacts/ – serialized inputs (CSV) and outputs (trained model.pkl, preprocessor.pkl, train/test splits)
• src/ – production code implementing the pipeline components (data_ingestion, data_transformation, model_trainer, predict_pipeline, train_pipeline), utilities, logger, exception handlers
• notebook/ – Jupyter notebooks and parallel source tree for EDA and exploratory pipeline code
• requirements.txt, setup.py – dependency and packaging metadata
• Separation of “pipeline” (train_pipeline.py, predict_pipeline.py) and “components” (ingestion/transformation/trainer), conforming to a layered pipeline architecture
Extract additional insights from the README
• Data source: Kaggle “Students Performance in Exams” CSV
• Models experimented with: Linear Regression, KNN, Decision Tree, XGBoost
• Hyperparameter tuning via Grid/Random Search
• Deployed as a Streamlit web app (hosted on Hugging Face Spaces)
• User inputs demographic + prior scores, UI triggers prediction and displays math score
• User Interface (Streamlit app in app.py)
• Prediction Service (predict_pipeline loading preprocessor.pkl & model.pkl)
• Training Pipeline (train_pipeline invoking data_ingestion → data_transformation → model_trainer → artifacts)
• Data Store (raw CSV dataset, train/test CSVs in artifacts/)
• Artifact Repository (serialized objects: model.pkl, preprocessor.pkl)
• Logging & Monitoring (logger.py, exception.py)
• External Services (Kaggle dataset source, hosting platform – e.g., Hugging Face Spaces)
b. Relationships and data flow:
1. Training flow (offline): raw data → DataIngestion → DataTransformation → ModelTrainer → serialized artifacts
2. Serving flow (online): user inputs → Streamlit app → PredictPipeline → loads preprocessor + model → transforms inputs → outputs prediction → UI display
3. Logging & exception paths intercept both flows for audit and error handling
c. Architectural patterns and design principles:
• Layered pipeline (clear separation between ingestion, transformation, and modeling)
• Modular components for testability and reuse (src/components)
• Single-page web UI pattern (Streamlit) with synchronous request-response for prediction
• Artifact versioning for reproducibility
d. Key technologies and frameworks:
• Python, pandas, scikit-learn (data/scaling)
• CatBoost (model training + evaluation)
• Streamlit (web interface)
• Logging module (custom logger.py)
• Packaging via setup.py, dependency control via requirements.txt