How adilmirza975/ml_projects works
1. Project type and purpose
- This is an end-to-end machine learning application for student performance prediction.
- The repository combines an offline training workflow with an online inference web app.
2. Two main execution paths
- Training path: prepares data, trains a model, and writes serialized artifacts.
- Serving path: loads the saved preprocessing/model artifacts and exposes predictions through a web UI.
- This separation is reflected in src/pipeline/train_pipeline.py vs. src/pipeline/predict_pipeline.py.
3. Core ML pipeline structure
- The ML logic is organized under src/components/:
- data_ingestion.py handles loading and splitting data.
- data_transformation.py handles feature preprocessing.
- model_trainer.py trains and evaluates the estimator.
- These are orchestrated by the pipeline layer, which keeps the components loosely coupled.
4. Data flow through training
- Raw dataset originates from the notebook data and/or ingested CSVs.
- Training produces intermediate datasets and final serialized objects in artifacts/.
- The presence of train.csv, test.csv, data.csv, preprocessor.pkl, and model.pkl shows a classic batch ML workflow with persisted preprocessing and model state.
5. Model artifact boundary
- artifacts/ is the key boundary between training and inference.
- The web app does not retrain; it depends on the persisted preprocessor.pkl and model.pkl.
- This makes the system deployable as a stateless prediction service after training.
6. Prediction/serving layer
- src/pipeline/predict_pipeline.py is the inference-side pipeline.
- It likely converts user input into model-ready features using the saved preprocessor, then produces predictions from the saved model.
- This is the bridge between the HTML form and the ML artifacts.
7. Web application layer
- app.py and application.py suggest a Flask-style entry point structure.
- templates/ contains the HTML views (home.html, index.html), and static/css/style.css provides UI styling.
- The web layer is primarily responsible for collecting user inputs and displaying prediction results, not training logic.
8. Deployment target
- .ebextensions/python.config indicates the project is configured for AWS Elastic Beanstalk deployment.
- This means the app is intended to run as a managed Python web service with environment configuration handled externally.
9. Supporting engineering utilities
- src/logger.py and src/exception.py provide cross-cutting concerns for observability and error handling.
- src/utils.py likely contains shared helper functions used by multiple pipeline stages.
- These modules support maintainability across both training and serving code paths.
10. Project packaging and dependencies
- setup.py and requirements.txt define packaging and Python dependencies.
- src/__init.py and package submodules organize the code as an importable application rather than a notebook-only prototype.
11. Notebook layer
- notebook/EDA STUDENT PERFORMANCE.ipynb and notebook/MODEL TRAINER.ipynb show the exploratory and experimental workflow used before formalizing the pipeline.
- These notebooks are not part of runtime execution, but they likely informed the final component design and model choice.
12. Training diagnostics
- catboost_info/ and similar notebook-local training outputs indicate CatBoost was used at some point or during experimentation.
- These directories are training diagnostics rather than runtime dependencies, so they belong to the model development side of the architecture.
13. High-level architecture summary
- The repository is a three-layer ML system:
- Data/model development: notebooks and training scripts
- Core ML backend: ingestion → transformation → model training
- Delivery layer: Flask web UI with saved artifacts for inference
- The central design principle is that training is offline, prediction is online, and the artifact directory is the contract between them.
Main entry points: train_pipeline.py Train Pipeline · app.py App · predict_pipeline.py Predict Pipeline · data_ingestion.py Ingestion
Components 26
Development
Notebooks and experiment outputs used to explore data and prototype the model.
- EDA Notebook · Explores the student performance dataset and informs feature/model choices.
notebook/EDA STUDENT PERFORMANCE.ipynb - Trainer Notebook · Notebook-based model development and training workflow.
notebook/MODEL TRAINER.ipynb - Sample Data · Notebook dataset used for exploration and experimentation.
notebook/data/stud.csv
Core ML
Reusable training and inference pipeline code.
- Train Pipeline · Coordinates the offline training workflow from raw data to saved artifacts.
src/pipeline/train_pipeline.py - Predict Pipeline · Loads saved preprocessing and model artifacts to serve predictions.
src/pipeline/predict_pipeline.py - Ingestion · Reads raw data and prepares train/test splits.
src/components/data_ingestion.py - Transformation · Builds and applies preprocessing for model-ready features.
src/components/data_transformation.py - Model Trainer · Trains and evaluates the estimator, then persists the model artifact.
src/components/model_trainer.py - Utils · Common helper functions reused across pipeline stages.
src/utils.py - Logger · Central logging support for the application and pipelines.
src/logger.py - Exceptions · Shared exception and error reporting utilities.
src/exception.py
Delivery
Web app entry points, templates, and static assets for serving predictions.
- App · Primary web application entry point for request handling.
app.py - WSGI App · Application bootstrap/launch module for the Flask-style service.
application.py - Templates · Rendered pages for the prediction UI.
templates/index.html - Home Page · Landing or form page for collecting user inputs.
templates/home.html - Stylesheet · Presentation styling for the web interface.
static/css/style.css
Deployment
Environment and packaging files for running the application.
Artifacts
Persisted datasets and serialized objects shared between training and inference.
- Artifacts · Shared boundary between offline training and online inference.
artifacts/model.pkl - Preprocessor · Saved preprocessing pipeline used during inference.
artifacts/preprocessor.pkl - Model · Trained prediction model loaded by the serving path.
artifacts/model.pkl - Train CSV · Persisted training split produced by ingestion.
artifacts/train.csv - Test CSV · Persisted test split produced by ingestion.
artifacts/test.csv - Data CSV · Persisted canonical dataset snapshot used in the pipeline.
artifacts/data.csv
Other components
- README · Top-level project documentation and usage context.
README.md
Connections 25
- EDA Notebook → Sample Data: explores · EDA uses the sample dataset for analysis.
- Trainer Notebook → Sample Data: trains on · Notebook training experiments consume the prepared dataset.
- Train Pipeline → Ingestion: orchestrates · Training pipeline starts with data ingestion.
- Train Pipeline → Transformation: orchestrates · Training pipeline applies feature transformation.
- Train Pipeline → Model Trainer: orchestrates · Training pipeline invokes model training.
- Ingestion → Data CSV: writes · Ingestion persists the canonical dataset snapshot.
- Ingestion → Train CSV: splits · Ingestion produces the training split artifact.
- Ingestion → Test CSV: splits · Ingestion produces the test split artifact.
- Transformation → Preprocessor: serializes · Transformation saves the fitted preprocessing pipeline.
- Model Trainer → Model: serializes · Model trainer writes the trained estimator artifact.
- Train Pipeline → Artifacts: publishes · Training pipeline outputs its results to the artifact boundary.
- Predict Pipeline → Preprocessor: loads · Inference pipeline restores the saved preprocessor.
- Predict Pipeline → Model: loads · Inference pipeline restores the trained model.
- App → Predict Pipeline: calls · Web request handling delegates prediction work to the inference pipeline.
- App → Templates: renders · Application renders HTML views for user interaction.
- Templates → Home Page: includes · Template set contains the home or input page.
- App → Stylesheet: serves · Application serves styling assets to the browser.
- WSGI App → App: boots · WSGI/application bootstrap initializes the web app.
- App → Logger: logs · Web layer uses shared logging support.
- Train Pipeline → Logger: logs · Training workflow uses shared logging support.
- Predict Pipeline → Exceptions: handles · Inference path uses shared error handling utilities.
- Train Pipeline → Utils: uses · Training workflow relies on shared helpers.
- Predict Pipeline → Utils: uses · Inference workflow relies on shared helpers.
- EB Config → App: deploys · Elastic Beanstalk configuration targets the web application.
- Setup → Deps: packages · Packaging and dependency metadata define the runtime environment.