# ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow architecture

> Architecture diagram of the GitHub repository ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow
- Repository: https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow

## Overview

##### Project Type and Purpose
- This is an end-to-end machine learning pipeline for binary classification (Pima Diabetes prediction) that emphasizes reproducibility (DVC), experiment tracking (MLflow), and modular stages (preprocess → train → evaluate).

##### Key Architectural Components
a. Version Control & Orchestration
- Git: source code versioning
- DVC: data & model versioning, pipeline orchestration (dvc.yaml / dvc.lock)
- Remote Storage: S3/GCS (or any DVC remote) for raw data, processed data, and model artifacts
b. Data Layer
- Raw data (data/raw/diabetes.csv)
- Processed data (data/processed/diabetes_clean.csv)
c. Pipeline Stages (src/)
- preprocess.py: reads raw data, cleans/transforms → writes processed CSV
- train.py: reads processed data, trains RandomForest → writes model.pkl
- evaluate.py: reads model & processed data → computes metrics
d. Experiment Tracking
- MLflow server/ UI: log hyperparameters, metrics, and artifacts per run
e. Parameters & Dependencies
- params.yaml: all hyperparameters (e.g., random_state, n_estimators, max_depth)
- requirements.txt: Python dependencies (scikit-learn, pandas, numpy, mlflow, dvc)

##### Relationships & Data Flow
- A directed pipeline DAG:
Raw Data ──(dvc)──> Preprocess ──> Clean Data ──> Train ──> Model Artifact ──> Evaluate
- How MLflow hooks into train & evaluate steps to log runs

##### Architectural Patterns & Design Principles
- Pipeline as Code: dvc.yaml defines stages, inputs, outputs, params
- Modularity: Each stage encapsulated in its own script under src/
- Reproducibility: Data + model versioned under DVC, parameters centralized in params.yaml
- Experiment Tracking: MLflow decoupled from code via API calls in train.py / evaluate.py

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Developer/CI
    Dev["Developer / CI"]:::developer

    %% Version Control & Orchestration
    subgraph "Version Control & Orchestration"
        GitRepo["Git Repository"]:::repo
        gitignore[".gitignore"]:::config
        dvcYAML["dvc.yaml"]:::config
        dvcLock["dvc.lock"]:::config
        dvcConfig[".dvc/config"]:::config
        DVCRemote["DVC Remote Storage"]:::cloud
    end

    Dev -->|"commit code"| GitRepo
    Dev -->|"dvc repro"| dvcYAML
    GitRepo ---|push/pull| DVCRemote
    GitRepo --> gitignore
    GitRepo --> dvcYAML
    GitRepo --> dvcLock
    GitRepo --> dvcConfig

    %% Data Layer
    subgraph "Versioned Artifacts"
        RawDir["Raw Data Directory"]:::storage
        RawFile["diabetes.csv.dvc"]:::storage
        ProcessedDir["Processed Data Directory"]:::storage
        Models["Model Artifacts"]:::storage
    end

    DVCRemote ---|stores raw data| RawDir
    RawDir --> RawFile
    RawFile -->|"dvc repro"| Preprocess

    DVCRemote ---|stores processed data| ProcessedDir
    DVCRemote ---|stores models| Models

    %% Pipeline Stages
    subgraph "Pipeline Stages"
        Preprocess["Preprocess Stage\nsrc/preprocess.py"]:::pipeline
        Train["Training Stage\nsrc/train.py"]:::pipeline
        Evaluate["Evaluation Stage\nsrc/evaluate.py"]:::pipeline
        Params["params.yaml"]:::config
        Requirements["requirements.txt"]:::config
    end

    RawFile -->|"preprocess: src/preprocess.py"| Preprocess
    Preprocess -->|"clean data"| ProcessedDir
    ProcessedDir -->|"train: src/train.py"| Train
    Train -->|"model.pkl"| Models
    Models -->|"evaluate: src/evaluate.py"| Evaluate

    Params --> Preprocess
    Params --> Train
    Requirements --> Preprocess
    Requirements --> Train
    Requirements --> Evaluate

    %% Experiment Tracking
    subgraph "Experiment Tracking"
        MLflow["MLflow Server / UI"]:::cloud
    end

    Train -->|"mlflow log run"| MLflow
    Evaluate -->|"mlflow log metrics"| MLflow
    Dev -->|"mlflow ui"| MLflow

    %% Documentation
    README["README.md"]:::config
    GitRepo --> README

    %% Click Events
    click GitRepo "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/blob/main/.gitignore"
    click gitignore "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/blob/main/.gitignore"
    click dvcYAML "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/blob/main/dvc.yaml"
    click dvcLock "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/blob/main/dvc.lock"
    click dvcConfig "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/tree/main/.dvc/config"
    click RawDir "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/tree/main/data/raw/"
    click RawFile "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/blob/main/data/raw/diabetes.csv.dvc"
    click ProcessedDir "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/tree/main/data/processed/"
    click Models "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/tree/main/models/"
    click Preprocess "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/blob/main/src/preprocess.py"
    click Train "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/blob/main/src/train.py"
    click Evaluate "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/blob/main/src/evaluate.py"
    click Params "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/blob/main/params.yaml"
    click Requirements "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/blob/main/requirements.txt"
    click README "https://github.com/ridabayi/end-to-end-machine-learning-pipeline-diabetes-prediction-with-dvc-mlflow/blob/main/README.md"

    %% Styles
    classDef pipeline fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#060,stroke-width:1px
    classDef cloud fill:#FFA500,stroke:#f60,stroke-width:1px
    classDef config fill:#FFD700,stroke:#aa0,stroke-width:1px
    classDef developer fill:#D3D3D3,stroke:#666,stroke-width:1px
    classDef repo fill:#EEE8AA,stroke:#990,stroke-width:1px
```
