# arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments architecture

> Architecture diagram of the GitHub repository arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments
- Repository: https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments

## Overview

##### Project type and purpose
- This is an end-to-end machine learning application for chicken disease classification.
- It includes: an MLOps pipeline (built with DVC), experiment tracking (MLflow + DagsHub), model training/inference code (Python modules under src/cnnClassifier), a simple Flask-based web UI (app.py + templates/index.html), and deployment automation (GitHub Actions → AWS ECR → EC2).

##### File structure
- Top‐level:
- config/ (YAML files for pipeline & params)
- dvc.yaml + params.yaml (defines DVC pipeline)
- main.py (CLI entry point to orchestrate pipelines)
- app.py + templates/index.html (Flask app for serving predictions)
- .github/workflows/cicd.yaml (CI/CD for building/pushing Docker & deploying to EC2)
- src/cnnClassifier/… (core components: data_ingestion, prepare_base_model, model_trainer, evaluation; configuration manager, entities, logger, pipelines)
- Research notebooks under research/ (exploratory steps, not part of prod system)
- setup.py, requirements.txt (packaging)

##### README for insights
- Describes the 4 major MLOps tools: DVC (for pipelines + data versioning), MLflow (experiment tracking), DAGsHub remote, GitHub Actions + AWS (CICD and deployment).
- Outlines the workflow: update configs → implement components → run dvc repro → monitor MLflow → push to production via GitHub Actions → EC2 self-hosted runner.

a. Main components:
1. Data Layer
- Raw data repository (e.g., local file system or remote storage tracked by DVC)
- DVC pipeline stages: data_ingestion → prepare_base_model → model_trainer → evaluation
2. Model Registry & Experiment Tracking
- MLflow server (remote MLFLOW_TRACKING_URI on DagsHub)
- MLflow UI (for metrics, parameters, artifacts)
3. Application Layer
- Flask Web App (app.py + index.html) exposing a /predict API
- Prediction pipeline (src/cnnClassifier/pipeline/predict.py) loading the trained model artifact and returning inference
4. CI/CD & Deployment
- GitHub Actions workflow (cicd.yaml)
- Docker build & push to AWS ECR
- EC2 instance (self-hosted runner) pulling the Docker image and running the container
b. Relationships & interactions:
- Developer commits → GitHub Actions triggers → builds Docker image → pushes to ECR → EC2 pulls image → runs container → exposes web app.
- During development, running “dvc repro” executes sequential pipeline stages; each stage logs metrics/artifacts to MLflow.
- Flask app loads the latest model artifact (from local volume or DVC cache) to serve predictions.
c. Architectural patterns & principles:
- Pipeline orchestration (DVC as a lightweight DAG orchestrator)
- Configuration-driven design (YAML + config manager)
- Separation of concerns: data ingestion, model prep, training, evaluation, inference
- MLOps integration: experiment tracking decoupled from compute; CI/CD automates build & deploy
d. Key technologies & libraries:
- Python 3.x, DVC, MLflow, Flask, GitHub Actions, AWS ECR/EC2, PyYAML, TensorFlow/Keras or PyTorch (implied by transfer learning), Docker

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CI/CD & Deployment
    subgraph "CI/CD & Deployment" 
        direction TB
        Dev["Developer Commit"]:::external
        GH["GitHub Actions Workflow"]:::deployment
        Docker["Docker Build & Push"]:::deployment
        ECR["AWS ECR (Container Registry)"]:::deployment
        EC2["EC2 Instance\n(Self-hosted Runner)"]:::deployment
        Dev -->|"push to repo"| GH
        GH -->|"build & push"| Docker
        Docker -->|"docker push"| ECR
        ECR -->|"docker pull"| EC2
        EC2 -->|"runs container"| AppContainer
    end

    %% Application Layer
    subgraph "Serving Environment (EC2 Container)"
        direction TB
        AppContainer["Flask App Container"]:::deployment
        FlaskApp["Flask Web App\n/app.py"]:::code
        Template["UI Template\n/index.html"]:::code
        PredictPipeline["Prediction Pipeline\n(predict.py)"]:::code
        ModelArtifact["Model Artifact\n(DVC/Docker Volume)"]:::storage
        User["User"]:::external
        AppContainer --> FlaskApp
        FlaskApp --> Template
        FlaskApp -->|"API request /predict"| PredictPipeline
        PredictPipeline -->|"load model"| ModelArtifact
        PredictPipeline -->|"response"| FlaskApp
        FlaskApp -->|"send response"| User
    end

    %% Training Environment
    subgraph "Training Environment"
        direction TB
        ConfigFiles["Global Config Files\n(config.yaml,\nparams.yaml)"]:::storage
        CLI["CLI Orchestrator\nmain.py"]:::code
        DVCDef["DVC Pipeline Definition\ndvc.yaml"]:::infra
        subgraph "DVC Pipeline Stages"
            direction TB
            Stage1["Data Ingestion\n(data_ingestion_pipeline.py)"]:::code
            Stage2["Base Model Prep\n(prepare_base_model_pipeline.py)"]:::code
            Stage3["Model Trainer\n(stage_03_model_trainer.py)"]:::code
            Stage4["Evaluation\n(stage_04_evaluation.py)"]:::code
            Stage1 --> Stage2
            Stage2 --> Stage3
            Stage3 --> Stage4
            Stage1 -->|"mlflow.log_*"| MLflow
            Stage2 -->|"mlflow.log_*"| MLflow
            Stage3 -->|"mlflow.log_*"| MLflow
            Stage4 -->|"mlflow.log_*"| MLflow
        end
        CLI -->|"dvc repro"| DVCDef
        DVCDef -->|"exec stages"| Stage1
        ConfigFiles --> Stage1
        ConfigFiles --> Stage2
        ConfigFiles --> Stage3
        ConfigFiles --> Stage4
    end

    %% MLOps Infrastructure
    subgraph "MLOps Infrastructure"
        direction TB
        DVCRemote["DVC Remote Storage\n(DVC Cache)"]:::storage
        MLflow["MLflow Server\n(DagsHub)"]:::infra
        DVCRemote -.->|"pull/push data"| Stage1
        DVCRemote -.->|"cache artifacts"| Stage2
        DVCRemote -.->|"cache artifacts"| Stage3
        DVCRemote -.->|"cache artifacts"| Stage4
    end

    %% Research Notebooks
    subgraph "Exploration (Research)"
        direction TB
        Notebooks["Research Notebooks\n(research/)"]:::external
        Notebooks --> ConfigFiles
        Notebooks --> Stage1
    end

    %% Click Events
    click Stage1 "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/src/cnnClassifier/pipeline/data_ingestion_pipeline.py"
    click Stage2 "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/src/cnnClassifier/pipeline/prepare_base_model_pipeline.py"
    click Stage3 "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/src/cnnClassifier/pipeline/stage_03_model_trainer.py"
    click Stage4 "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/src/cnnClassifier/pipeline/stage_04_evaluation.py"
    click PredictPipeline "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/src/cnnClassifier/pipeline/predict.py"
    click Dev "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/."
    click GH "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/.github/workflows/cicd.yaml"
    click AppContainer "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/app.py"
    click Template "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/templates/index.html"
    click ConfigFiles "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/config/config.yaml"
    click CLI "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/main.py"
    click DVCDef "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/dvc.yaml"
    click Notebooks "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/tree/main/research/"
    click DVCRemote "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/."
    click MLflow "https://github.com/arqamansari23/end-to-end-deep-learning-project-using-mlops-dvc-pipeline-with-deployments/blob/main/."
    
    %% Classes
    classDef code fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef infra fill:#90EE90,stroke:#333,stroke-width:1px
    classDef deployment fill:#FFA500,stroke:#333,stroke-width:1px
    classDef storage fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef external fill:#f0f0f0,stroke:#666,stroke-dasharray: 2 2,stroke-width:1px
```
