How ayushmittal62/end-to-end-ml works
Project Type & Purpose
• This is a full-stack, end-to-end ML application for “Student Performance Prediction.”
• It contains:
• Data pipelines (ingestion, transformation, training) under src/
• A Flask-based serving layer (application.py + templates/) exposing both HTML form and JSON prediction endpoints
• Pretrained artifacts in artifacts/
• Experiment logs and CatBoost training metadata in logs/ and catboost_info/
• Dockerization/CI-CD (Procfile, .ebextensions for AWS Elastic Beanstalk, GitHub Actions referenced in README)
Key Components
a. Data Sources & Storage
• Raw CSVs: notebook/data/stud.csv; artifacts/train.csv, test.csv
• Model artifacts: artifacts/model.pkl, artifacts/preprocessor.pkl
• Experiment logs: logs/*.log; catboost_info/ (TensorBoard events, training JSON)
b. Pipeline Modules (src/)
• components/
• data_ingestion.py ➔ reads raw CSVs
• data_transformation.py ➔ feature engineering, scaling, encodings
• model_trainer.py ➔ trains & evaluates the model, serializes artifacts
• pipeline/
• train_pipeline.py ➔ orchestration of ingestion→transformation→training
• predict_pipeline.py ➔ loads artifacts and runs inference
• utils.py, exception.py, logger.py support common functions & logging
c. Serving Layer
• application.py ➔ Flask app entrypoint
• templates/home.html + index.html ➔ UI for form-based predictions
• Static JSON endpoint for programmatic consumption
d. Infrastructure & Deployment
• Dockerfile (implied by README/setup.py) & Procfile for container startup
• AWS Elastic Beanstalk config in .ebextensions/python.config
• CI/CD orchestrated via GitHub Actions → builds Docker image → pushes to AWS ECR → deploys to EC2/EB
• MLflow or similar tracking (not code-visible but mentioned in README)
Relationships & Data Flow
• [Raw Data (CSV)] → Data Ingestion → Data Transformation → Model Trainer → [Artifacts (model.pkl + preprocessor.pkl)]
→ saved to artifacts/ and optionally to a model registry
• Serving:
• application.py loads artifacts at startup
• Client (browser or API) → Flask endpoint → Predict Pipeline → returns scores/back to client
• Deployment:
• Code pushed to GitHub → GitHub Actions → Docker build → ECR → EC2/EB → running container logs to logs/ and catboost_info/
Architectural Patterns & Principles
• Separation of Concerns: ingestion, transform, training, prediction split into distinct modules
• Pipeline Orchestration: train_pipeline & predict_pipeline classes manage workflow steps
• 3-Tier (UI / Business Logic (Flask + Pipelines) / Data Layer (artifacts, file storage, logs))
• Infrastructure-as-Code: .ebextensions, Procfile, setup.py for repeatable deployment