# vaibhavsingh6952/mlpipeline architecture

> Architecture diagram of the GitHub repository vaibhavsingh6952/mlpipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vaibhavsingh6952/mlpipeline
- Repository: https://github.com/vaibhavsingh6952/mlpipeline

## Overview

##### Project Type and Purpose
- This is a data-engineering/machine-learning pipeline that uses:
- DVC for data and model versioning
- MLflow for experiment tracking
- It orchestrates four main stages on the Pima Indians Diabetes dataset: data ingestion, preprocessing, training, and evaluation.

##### File Structure
- Top-level:
- dvc.yaml, dvc.lock, .dvc/ → DVC pipeline definitions & config
- data/raw, data/processed → raw and processed datasets under DVC control
- models/ → stored trained models
- src/ → pipeline scripts (preprocess.py, train.py, evaluate.py, evaluate.py)
- params.yaml → pipeline hyperparameters
- requirements.txt → Python dependencies
- The presence of dvc.yaml indicates a DAG of pipeline stages.

##### Extract Insights from README
- Confirms DVC pipeline + MLflow integration.
- Two experiments are logged to DagsHub: one for training, one for evaluation.
- Data artifacts (raw and processed) and model artifacts are versioned in DagsHub.

##### Main Components
a. Data Stores
- Raw Data Store (data/raw, DVC remote)
- Processed Data Store (data/processed, DVC remote)
- Model Artifact Store (models/, DVC remote)
b. Compute/Processing Steps
- Preprocessing (src/preprocess.py)
- Training (src/train.py, RF classifier)
- Evaluation (src/evaluate.py)
c. Orchestration & Versioning
- DVC Pipeline (dvc.yaml)
- params.yaml for hyperparameters
d. Experiment Tracking
- MLflow Tracking Server (hosted on DagsHub)
e. External Services
- DVC Remote Storage on DagsHub
- MLflow Server on DagsHub

##### Relationships & Data Flow
- At each Compute step (Training & Evaluation), send metrics/logs to MLflow Tracking Server
- DVC pipeline orchestrates these steps, tracking inputs/outputs in dvc.lock

##### Architectural Patterns and Principles
- Pipeline as DAG (defined by dvc.yaml)
- Data versioning and reproducibility via DVC
- Experiment tracking decoupled from compute via MLflow
- Parameterization via params.yaml to enable configurable runs

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        dvc["DVC Pipeline (dvc.yaml)"]:::compute
        lock["DVC Lock File (dvc.lock)"]:::compute
    end

    %% Local Compute & Scripts
    subgraph "Local Compute & Scripts"
        direction TB
        Preprocess["Preprocess Script"]:::compute
        Train["Training Script"]:::compute
        Evaluate["Evaluation Script"]:::compute
        params["Hyperparameters (params.yaml)"]:::doc
        req["Dependencies (requirements.txt)"]:::doc
    end

    %% Data & Model Stores
    subgraph "Data & Model Stores"
        direction TB
        RawData[(Raw Data Store)]:::datastore
        ProcessedData[(Processed Data Store)]:::datastore
        ModelStore[(Model Artifact Store)]:::datastore
    end

    %% Remote Services
    subgraph "Remote Services"
        direction TB
        DVCRemote["DVC Remote (DagsHub)"]:::cloud
        MLflow["MLflow Tracking Server"]:::cloud
    end

    %% Data Flow
    RawData -->|raw data| Preprocess
    Preprocess -->|processed data| ProcessedData
    ProcessedData -->|training data| Train
    params -->|hyperparameters| Train
    Train -->|model artifacts| ModelStore
    ProcessedData -->|test data| Evaluate
    ModelStore -->|model| Evaluate

    %% Experiment Logging
    Train -.->|logs metrics| MLflow
    Evaluate -.->|logs metrics| MLflow

    %% Orchestration Control
    dvc ---|orchestrates| Preprocess
    dvc ---|orchestrates| Train
    dvc ---|orchestrates| Evaluate

    %% Versioning Control
    dvc ---|tracks| lock
    dvc ---|uses remote config| DVCRemote

    %% Click Events
    click RawData "https://github.com/vaibhavsingh6952/mlpipeline/tree/master/data/raw/"
    click ProcessedData "https://github.com/vaibhavsingh6952/mlpipeline/tree/master/data/processed/"
    click ModelStore "https://github.com/vaibhavsingh6952/mlpipeline/tree/master/models/"
    click Preprocess "https://github.com/vaibhavsingh6952/mlpipeline/blob/master/src/preprocess.py"
    click Train "https://github.com/vaibhavsingh6952/mlpipeline/blob/master/src/train.py"
    click Evaluate "https://github.com/vaibhavsingh6952/mlpipeline/blob/master/src/evaluate.py"
    click dvc "https://github.com/vaibhavsingh6952/mlpipeline/blob/master/dvc.yaml"
    click lock "https://github.com/vaibhavsingh6952/mlpipeline/blob/master/dvc.lock"
    click params "https://github.com/vaibhavsingh6952/mlpipeline/blob/master/params.yaml"
    click DVCRemote "https://github.com/vaibhavsingh6952/mlpipeline/tree/master/.dvc/config"
    click req "https://github.com/vaibhavsingh6952/mlpipeline/blob/master/requirements.txt"

    %% Styles
    classDef datastore fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef compute fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef cloud fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef doc fill:#EEE,stroke:#000,stroke-width:1px;
```
