# lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub architecture

> Architecture diagram of the GitHub repository lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub
- Repository: https://github.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub

## Overview

##### Project Type & Purpose
- Type: A machine learning pipeline that leverages DVC for data/model version control and MLflow for experiment tracking.
- Purpose: To preprocess a dataset, train a Random Forest Classifier (on the Pima Indians Diabetes Dataset), and evaluate model performance while tracking all experiments, parameters, and artifacts.

##### Analyzing the File Structure
- Top-level files and directories include:
- DVC-related files (dvc.yaml, dvc.lock, .dvc/ directory, params.yaml): Indicating that the project’s data and model lifecycle is managed by DVC.
- Data directories (data/raw, data/processed): Raw data is input and the processed output is stored in a dedicated location.
- Models directory (models): The trained model file(s) will be saved here.
- Source code directory (src): Contains key Python scripts (preprocess.py, train.py, evaluate.py) representing the pipeline stages.
- This structure highlights a clear separation of concerns – data management, model storage, and code/scripts – which is common for machine learning pipelines.

##### Insights from the README
- The README explains the three core pipeline stages:
- Preprocessing: Reads raw data and performs transformation (reading from data/raw/data.csv and outputting to data/processed/data.csv).
- Training: Uses preprocessed data to train a Random Forest Classifier and saves the model (logged in MLflow and stored under models).
- Evaluation: Evaluates the model, logging metrics back to MLflow.
- Both DVC and MLflow are central to the workflow: DVC guarantees reproducibility and proper versioning while MLflow provides a tracking mechanism for experiments and parameters.

A. Identify Main Components:
- Data Storage:
- Raw Data (data/raw/data.csv)
- Processed Data (data/processed/data.csv)
- Pipeline Stages (as distinct components):
- Preprocessing (src/preprocess.py)
- Training (src/train.py)
- Evaluation (src/evaluate.py)
- Model Artifacts:
- Trained model file (models/random_forest.pkl or similar)
- Experiment & Versioning Tools:
- DVC: Represented by config files (dvc.yaml, dvc.lock, params.yaml, .dvc folder) that manage stage dependencies and track the data and model versions.
- MLflow: Depicted as an external service or module that logs (parameters, metrics, artifacts) from training and evaluation stages.

B. Define Relationships & Interactions:
- Data Flow:
- Raw Data flows into the Preprocessing stage.
- Preprocessed Data flows from the Preprocessing stage into the Training stage.
- The Training stage outputs a trained model stored in the Models component.
- The Evaluation stage uses the trained model (from the Models directory) and possibly the data to compute and log evaluation metrics.
- Control Flow & Versioning:
- DVC acts as an orchestration tool to maintain pipeline stage dependencies and triggers re-execution when inputs (data, params, scripts) change.
- MLflow is used in both the Training and Evaluation components to record hyperparameters and metrics.

C. Highlight Architectural Patterns and Principles:
- Modular Pipeline: Each stage (preprocessing, training, evaluation) is implemented as a separate module, promoting separation of concerns.
- Reproducibility and Version Control: Use of DVC ensures that each stage is reproducible. Parameters and dependencies are versioned alongside code.
- Experiment Tracking: MLflow integration for capturing experiment details drives iterative improvements.

D. Relevant Technologies/Frameworks:
- Python: The main programming language used in all pipeline stages.
- DVC: For orchestrating reproducible pipeline stages and versioning datasets/models.
- MLflow: For logging and tracking experiments.
- Scikit-learn: For model training (Random Forest Classifier).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Version Control & Experiment Tracking Subgraph
    subgraph "Version Control & Experiment Tracking"
        subgraph "DVC Files"
            DVC_yaml["dvc.yaml"]:::orchestrator
            DVC_lock["dvc.lock"]:::orchestrator
            DVC_params["params.yaml"]:::orchestrator
            DVC_dir[".dvc/"]:::orchestrator
        end
        DVC_aggr["DVC Orchestration"]:::orchestrator
        MLflow["MLflow Tracking"]:::external
    end

    %% Connect DVC file nodes to the aggregator
    DVC_yaml --> DVC_aggr
    DVC_lock --> DVC_aggr
    DVC_params --> DVC_aggr
    DVC_dir --> DVC_aggr

    %% Machine Learning Pipeline Subgraph
    subgraph "Machine Learning Pipeline"
        RD["Raw Data"]:::data
        PP["Preprocessing Stage"]:::processing
        PD["Processed Data"]:::data
        TR["Training Stage"]:::processing
        MA["Model Artifacts Storage"]:::data
        EV["Evaluation Stage"]:::processing
    end

    %% Data Flow within the Pipeline
    RD -->|"reads"| PP
    PP -->|"transforms"| PD
    PD -->|"trains"| TR
    TR -->|"produces"| MA
    MA -->|"evaluates"| EV

    %% DVC Orchestration triggering pipeline stages (dashed arrows)
    DVC_aggr ---|"orchestrates"| PP
    DVC_aggr ---|"orchestrates"| TR
    DVC_aggr ---|"orchestrates"| EV

    %% Logging connections to MLflow
    TR -->|"logs"| MLflow
    EV -->|"logs"| MLflow

    %% Click Events for Component Mappings
    click RD "https://github.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub/blob/main/data/raw/data.csv"
    click PD "https://github.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub/blob/main/data/processed/data.csv"
    click PP "https://github.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub/blob/main/src/preprocess.py"
    click TR "https://github.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub/blob/main/src/train.py"
    click EV "https://github.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub/blob/main/src/evaluate.py"
    click MA "https://github.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub/tree/main/models/"
    click DVC_yaml "https://github.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub/blob/main/dvc.yaml"
    click DVC_lock "https://github.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub/blob/main/dvc.lock"
    click DVC_params "https://github.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub/blob/main/params.yaml"
    click DVC_dir "https://github.com/lalit-rg/end-to-end-diabetes-prediction-using-mlflow-dagshub/tree/main/.dvc/"

    %% Styles
    classDef orchestrator fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef external fill:#A9DFBF,stroke:#27AE60,stroke-width:2px;
    classDef processing fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef data fill:#D5F5E3,stroke:#196F3D,stroke-width:2px;
```
