# ayushmittal62/end-to-end-ml architecture

> Architecture diagram of the GitHub repository ayushmittal62/end-to-end-ml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayushmittal62/end-to-end-ml
- Repository: https://github.com/ayushmittal62/end-to-end-ml

## Overview

##### Project Type & Purpose
- This is a full-stack, end-to-end ML application for “Student Performance Prediction.”
- It contains:
- Data pipelines (ingestion, transformation, training) under src/
- A Flask-based serving layer (application.py + templates/) exposing both HTML form and JSON prediction endpoints
- Pretrained artifacts in artifacts/
- Experiment logs and CatBoost training metadata in logs/ and catboost_info/
- Dockerization/CI-CD (Procfile, .ebextensions for AWS Elastic Beanstalk, GitHub Actions referenced in README)

##### Key Components
a. Data Sources & Storage
- Raw CSVs: notebook/data/stud.csv; artifacts/train.csv, test.csv
- Model artifacts: artifacts/model.pkl, artifacts/preprocessor.pkl
- Experiment logs: logs/*.log; catboost_info/ (TensorBoard events, training JSON)
b. Pipeline Modules (src/)
- components/
- data_ingestion.py ➔ reads raw CSVs
- data_transformation.py ➔ feature engineering, scaling, encodings
- model_trainer.py ➔ trains & evaluates the model, serializes artifacts
- pipeline/
- train_pipeline.py ➔ orchestration of ingestion→transformation→training
- predict_pipeline.py ➔ loads artifacts and runs inference
- utils.py, exception.py, logger.py support common functions & logging
c. Serving Layer
- application.py ➔ Flask app entrypoint
- templates/home.html + index.html ➔ UI for form-based predictions
- Static JSON endpoint for programmatic consumption
d. Infrastructure & Deployment
- Dockerfile (implied by README/setup.py) & Procfile for container startup
- AWS Elastic Beanstalk config in .ebextensions/python.config
- CI/CD orchestrated via GitHub Actions → builds Docker image → pushes to AWS ECR → deploys to EC2/EB
- MLflow or similar tracking (not code-visible but mentioned in README)

##### Relationships & Data Flow
- [Raw Data (CSV)] → Data Ingestion → Data Transformation → Model Trainer → [Artifacts (model.pkl + preprocessor.pkl)]
→ saved to artifacts/ and optionally to a model registry
- Serving:
- application.py loads artifacts at startup
- Client (browser or API) → Flask endpoint → Predict Pipeline → returns scores/back to client
- Deployment:
- Code pushed to GitHub → GitHub Actions → Docker build → ECR → EC2/EB → running container logs to logs/ and catboost_info/

##### Architectural Patterns & Principles
- Separation of Concerns: ingestion, transform, training, prediction split into distinct modules
- Pipeline Orchestration: train_pipeline & predict_pipeline classes manage workflow steps
- 3-Tier (UI / Business Logic (Flask + Pipelines) / Data Layer (artifacts, file storage, logs))
- Infrastructure-as-Code: .ebextensions, Procfile, setup.py for repeatable deployment

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        BrowserUI["Web Browser UI"]:::ui
        APIClient["API Client"]:::ui
    end

    %% Serving Layer
    subgraph "Serving Layer"
        FlaskApp["Flask App\n(application.py)"]:::service
        Templates["HTML Templates\n(home.html, index.html)"]:::service
    end

    %% ML Pipeline Layer
    subgraph "ML Pipeline" 
        TrainPipeline["train_pipeline.py"]:::pipeline
        DataIngestion["data_ingestion.py"]:::pipeline
        DataTransformation["data_transformation.py"]:::pipeline
        ModelTrainer["model_trainer.py"]:::pipeline
        PredictPipeline["predict_pipeline.py"]:::pipeline
        Utils["utils.py"]:::pipeline
        ExceptionMod["exception.py"]:::pipeline
        LoggerMod["logger.py"]:::pipeline
    end

    %% Data Layer
    subgraph "Data Layer"
        RawCSV["Raw CSV Data\nstud.csv"]:::storage
        Artifacts["Artifacts Storage\nmodel.pkl, preprocessor.pkl\ntest/train/data.csv"]:::storage
        Logs["Experiment Logs\nlogs/"]:::storage
        CatBoostMeta["CatBoost Metadata\ncatboost_info/"]:::storage
    end

    %% Infrastructure & Deployment
    subgraph "Infrastructure & Deployment"
        GitHub["GitHub Actions CI/CD"]:::infra
        DockerBuild["Docker Build"]:::infra
        ECR["AWS ECR"]:::infra
        EC2EB["AWS EC2 / Elastic Beanstalk"]:::infra
        MLflow["MLflow Tracking Server"]:::external
    end

    %% Data Flow
    RawCSV -->|ingest| DataIngestion
    DataIngestion -->|transform| DataTransformation
    DataTransformation -->|train| ModelTrainer
    ModelTrainer -->|save artifacts| Artifacts
    ModelTrainer -->|log metrics| Logs
    ModelTrainer -->|log metadata| CatBoostMeta
    ModelTrainer -->|track| MLflow

    %% Serving Flow
    BrowserUI -->|HTTP Request| FlaskApp
    APIClient -->|API Call| FlaskApp
    FlaskApp -->|load artifacts| Artifacts
    FlaskApp -->|call predict| PredictPipeline
    PredictPipeline -->|read artifacts| Artifacts
    PredictPipeline -->|return prediction| FlaskApp
    FlaskApp -->|response| BrowserUI

    %% CI/CD Flow
    GitHub -->|on push| DockerBuild
    DockerBuild -->|push image| ECR
    ECR -->|deploy| EC2EB
    EC2EB -->|runs container| FlaskApp

    %% Pipeline Orchestration
    TrainPipeline --> DataIngestion
    TrainPipeline --> DataTransformation
    TrainPipeline --> ModelTrainer
    PredictPipeline --> Utils
    PredictPipeline --> ExceptionMod
    PredictPipeline --> LoggerMod

    %% Click Events
    click DataIngestion "https://github.com/ayushmittal62/end-to-end-ml/blob/main/src/components/data_ingestion.py"
    click DataTransformation "https://github.com/ayushmittal62/end-to-end-ml/blob/main/src/components/data_transformation.py"
    click ModelTrainer "https://github.com/ayushmittal62/end-to-end-ml/blob/main/src/components/model_trainer.py"
    click TrainPipeline "https://github.com/ayushmittal62/end-to-end-ml/blob/main/src/pipeline/train_pipeline.py"
    click PredictPipeline "https://github.com/ayushmittal62/end-to-end-ml/blob/main/src/pipeline/predict_pipeline.py"
    click Utils "https://github.com/ayushmittal62/end-to-end-ml/blob/main/src/utils.py"
    click ExceptionMod "https://github.com/ayushmittal62/end-to-end-ml/blob/main/src/exception.py"
    click LoggerMod "https://github.com/ayushmittal62/end-to-end-ml/blob/main/src/logger.py"
    click FlaskApp "https://github.com/ayushmittal62/end-to-end-ml/blob/main/application.py"
    click Templates "https://github.com/ayushmittal62/end-to-end-ml/blob/main/templates/home.html"
    click Templates "https://github.com/ayushmittal62/end-to-end-ml/blob/main/templates/index.html"
    click RawCSV "https://github.com/ayushmittal62/end-to-end-ml/blob/main/notebook/data/stud.csv"
    click Artifacts "https://github.com/ayushmittal62/end-to-end-ml/blob/main/artifacts/model.pkl"
    click Artifacts "https://github.com/ayushmittal62/end-to-end-ml/blob/main/artifacts/preprocessor.pkl"
    click Artifacts "https://github.com/ayushmittal62/end-to-end-ml/blob/main/artifacts/train.csv"
    click Artifacts "https://github.com/ayushmittal62/end-to-end-ml/blob/main/artifacts/test.csv"
    click Artifacts "https://github.com/ayushmittal62/end-to-end-ml/blob/main/artifacts/data.csv"
    click Logs "https://github.com/ayushmittal62/end-to-end-ml/tree/main/logs/"
    click CatBoostMeta "https://github.com/ayushmittal62/end-to-end-ml/tree/main/catboost_info/"
    click GitHub "https://github.com/ayushmittal62/end-to-end-ml/tree/main/.github/workflows"
    click DockerBuild "https://github.com/ayushmittal62/end-to-end-ml/tree/main/Dockerfile"
    click EC2EB "https://github.com/ayushmittal62/end-to-end-ml/blob/main/.ebextensions/python.config"
    click EC2EB "https://github.com/ayushmittal62/end-to-end-ml/tree/main/Procfile"
    click EC2EB "https://github.com/ayushmittal62/end-to-end-ml/blob/main/setup.py"
    click EC2EB "https://github.com/ayushmittal62/end-to-end-ml/blob/main/requirements.txt"

    %% Styles
    classDef ui fill:#CCF,stroke:#333,stroke-width:1px
    classDef service fill:#CFF,stroke:#333,stroke-width:1px
    classDef pipeline fill:#CFC,stroke:#333,stroke-width:1px
    classDef storage fill:#FCC,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef infra fill:#FCF,stroke:#333,stroke-width:1px
    classDef external fill:#FFC,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2
```
