# from-kishore-kumar/end-to-end-ml-project-with-mlflow architecture

> Architecture diagram of the GitHub repository from-kishore-kumar/end-to-end-ml-project-with-mlflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow
- Repository: https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow

## Overview

##### Project type and purpose
- This is an end-to-end machine-learning application and web service for wine-quality prediction.
- It implements a reusable ML pipeline (data ingestion, validation, transformation, training, evaluation), experiment tracking (MLflow, DagsHub), and a Flask-based web front end. It also has CI/CD via GitHub Actions and Docker/AWS deployment.

##### File structure
- Top level:
- app.py: Flask application
- main.py: orchestrates full pipeline run
- config/: YAML configuration
- params.yaml, schema.yaml: param and schema definitions
- Dockerfile, .github/workflows/main.yaml: container build & CI/CD
- research/: exploratory notebooks
- src/mlProject/: core pipeline code organized by components, config, constants, entities, pipeline stages, and utils
- templates/: HTML templates for the Flask UI
- Patterns: clear pipeline stages (stage_01…05), modular components under src/mlProject/components.

##### README for additional insights
- Describes MLflow tracking, AWS ECR/ECS deployment, Flask UI, and CI/CD steps.
- Workflow: update configs → run pipeline → serve model via Flask → monitor with MLflow & DagsHub.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "Data Sources"
        RawData["Raw Data"]:::storage
        ConfigFiles["Config & Schema"]:::storage
    end

    click ConfigFiles "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/config/config.yaml"
    click ConfigFiles "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/params.yaml"
    click ConfigFiles "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/schema.yaml"

    %% Configuration Manager
    ConfigMgr["Configuration Manager"]:::service
    click ConfigMgr "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/config/configuration.py"

    %% Pipeline Orchestrator
    Orchestrator["Pipeline Orchestrator\n(main.py)"]:::service
    click Orchestrator "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/main.py"

    %% Pipeline Stages
    subgraph "ML Pipeline (Stages)" 
        direction TB
        Ingestion["Stage 1: Data Ingestion"]:::pipeline
        Validation["Stage 2: Data Validation"]:::pipeline
        Transformation["Stage 3: Data Transformation"]:::pipeline
        Trainer["Stage 4: Model Trainer"]:::pipeline
        Evaluation["Stage 5: Model Evaluation"]:::pipeline
        Prediction["Prediction/Inference"]:::pipeline
    end

    click Ingestion "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/pipeline/stage_01_data_ingestion.py"
    click Validation "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/pipeline/stage_02_data_validation.py"
    click Transformation "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/pipeline/stage_03_data_transformation.py"
    click Trainer "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/pipeline/stage_04_model_trainer.py"
    click Evaluation "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/pipeline/stage_05_model_evaluation.py"
    click Prediction "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/pipeline/prediction.py"

    %% Component Implementations
    subgraph "Component Logic"
        DataIngestComp["Data Ingestion Logic"]:::pipeline
        DataValidateComp["Data Validation Logic"]:::pipeline
        DataTransformComp["Data Transformation Logic"]:::pipeline
        ModelTrainComp["Model Trainer Logic"]:::pipeline
        ModelEvalComp["Model Evaluation Logic"]:::pipeline
    end

    click DataIngestComp "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/components/data_ingestion.py"
    click DataValidateComp "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/components/data_validation.py"
    click DataTransformComp "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/components/data_transformation.py"
    click ModelTrainComp "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/components/model_trainer.py"
    click ModelEvalComp "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/components/model_evaluation.py"
    click EntityDef "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/entity/config_entity.py"

    %% Utilities
    Utils["Utility Functions"]:::service
    click Utils "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/src/mlProject/utils/common.py"

    %% Experiment Tracking & Registry
    MLflow["MLflow / DagsHub"]:::service

    %% Model Artifact Store
    ModelStore["Model Artifact Store\n(local/S3)"]:::storage

    %% Flask Web App
    subgraph "Flask Web Application"
        FlaskApp["Flask App (app.py)"]:::service
        Templates["HTML Templates"]:::ui
    end

    click FlaskApp "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/app.py"
    click Templates "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/templates/index.html"
    click Templates "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/templates/results.html"

    %% CI/CD & Deployment
    subgraph "CI/CD & Deployment"
        GitHubActions["GitHub Actions"]:::service
        DockerImage["Dockerfile"]:::infra
        AWSInfra["AWS ECR & EC2"]:::infra
    end

    click GitHubActions "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/.github/workflows/main.yaml"
    click DockerImage "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/tree/main/Dockerfile"

    %% Research Notebooks
    subgraph "Exploratory Research"
        direction TB
        NB1["data_ingestion.ipynb"]:::service
        NB2["data_validation.ipynb"]:::service
        NB3["data_transformation.ipynb"]:::service
        NB4["model_trainer.ipynb"]:::service
        NB5["model_evaluation.ipynb"]:::service
        NB6["trials.ipynb"]:::service
    end

    click NB1 "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/research/data_ingestion.ipynb"
    click NB2 "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/research/data_validation.ipynb"
    click NB3 "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/research/data_transformation.ipynb"
    click NB4 "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/research/model_trainer.ipynb"
    click NB5 "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/research/model_evaluation.ipynb"
    click NB6 "https://github.com/from-kishore-kumar/end-to-end-ml-project-with-mlflow/blob/main/research/trials.ipynb"

    %% Connections
    RawData --> Ingestion
    ConfigFiles --> Ingestion
    ConfigFiles --> Validation
    ConfigFiles --> Transformation

    Ingestion --> DataIngestComp --> Validation
    Validation --> DataValidateComp --> Transformation
    Transformation --> DataTransformComp --> Trainer
    Trainer --> ModelTrainComp --> Evaluation
    Evaluation --> ModelEvalComp --> Prediction

    Orchestrator --> Ingestion
    ConfigMgr --> Orchestrator

    Prediction -->|"logs metrics, params, artifacts"| MLflow
    Prediction -->|saves model| ModelStore

    MLflow -->|"serves UI"| MLflow
    FlaskApp -->|loads model| ModelStore
    FlaskApp -->|uses config| ConfigMgr
    FlaskApp -->|"HTTP requests"| Prediction

    GitHubActions --> DockerImage --> AWSInfra
    GitHubActions --> AWSInfra

    %% Styles
    classDef pipeline fill:#D0E6FF,stroke:#2B6CB0
    classDef service fill:#D1F7C4,stroke:#2F855A
    classDef infra fill:#FEEBC8,stroke:#C05621
    classDef storage fill:#F0FFF4,stroke:#276749
    classDef ui fill:#FAF5FF,stroke:#6B46C1
```
