# larrysman/machine_learning_pipeline_with_dvc architecture

> Architecture diagram of the GitHub repository larrysman/machine_learning_pipeline_with_dvc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/larrysman/machine_learning_pipeline_with_dvc
- Repository: https://github.com/larrysman/machine_learning_pipeline_with_dvc

## Overview

##### Project type and purpose
- This is an end-to-end, modular machine-learning pipeline project built in Python. It uses DVC to define, execute, and version each stage of the data workflow (a DAG of ETL → feature engineering → model training/evaluation → deployment). Git tracks code and DVC tracks data, models, and metrics.

##### File structure
- Top-level:
- `.dvc/`, `.dvcignore`, `dvc.yaml` & `dvc.lock`: DVC configuration and pipeline definition files.
- `requirements.txt`: Python dependencies.
- `src/`: six single-responsibility Python scripts, each implementing one pipeline stage (data_collection, data_preparation, feature_engineering, data_preprocessing, model_building, model_evaluation).
- No traditional “frontend/backend” split—this is a data-processing pipeline. The core components are the code modules, the DVC orchestration layer, the raw/intermediate data store, the model artifacts store, and the metrics/artifacts output.

##### README for additional insights
- External services: local DVC remote (`dvcstore`), but could be S3/GDrive/Azure in other contexts.

##### Detail without over-splitting
- Of the six pipeline stages as individual components.
- DVC’s two roles: orchestration (reading `dvc.yaml` → running stages) and artifact versioning (push/pull to remote).
- Feedback loop if a stage output changes and the cache invalidates downstream stages.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Version Control
    subgraph "Version Control"
        README["README.md"]:::docs
        Req["requirements.txt"]:::docs
        GitIgnore[".gitignore"]:::docs
        click README "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/README.md"
        click Req "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/requirements.txt"
        click GitIgnore "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/.gitignore"
    end

    %% DVC Metadata
    subgraph "DVC Metadata"
        DVCConf[".dvc/"]:::dvcmeta
        DVCY["dvc.yaml"]:::dvcmeta
        DVCLock["dvc.lock"]:::dvcmeta
        DVCIgnore[".dvcignore"]:::dvcmeta
        click DVCConf "https://github.com/larrysman/machine_learning_pipeline_with_dvc/tree/main/.dvc/"
        click DVCY "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/dvc.yaml"
        click DVCLock "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/dvc.lock"
        click DVCIgnore "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/.dvcignore"
    end

    %% Orchestrator
    Orchestrator["DVC Orchestrator"]:::orchestrator

    %% Pipeline Stages
    DC["Data Collection\n(src/data_collection.py)"]:::module
    DP["Data Preparation\n(src/data_preparation.py)"]:::module
    FE["Feature Engineering\n(src/feature_engineering.py)"]:::module
    DPr["Data Preprocessing\n(src/data_preprocessing.py)"]:::module
    MB["Model Building\n(src/model_building.py)"]:::module
    ME["Model Evaluation\n(src/model_evaluation.py)"]:::module

    click DC "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/src/data_collection.py"
    click DP "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/src/data_preparation.py"
    click FE "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/src/feature_engineering.py"
    click DPr "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/src/data_preprocessing.py"
    click MB "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/src/model_building.py"
    click ME "https://github.com/larrysman/machine_learning_pipeline_with_dvc/blob/main/src/model_evaluation.py"

    %% Local Storage
    subgraph "Local Storage"
        Raw["Raw Data"]:::data
        Int["Intermediate Data"]:::data
        Models["Model Artifacts"]:::data
        Metrics["Evaluation Outputs"]:::data
    end

    %% Remote Storage
    DVCRemote["DVC Remote Storage"]:::remote

    %% Relationships
    README -->|version control| DVCY
    Req -->|tracks dependencies| DVCY
    GitIgnore -.-> DVCIgnore
    DVCY -->|pipeline definition| Orchestrator
    DVCLock -->|state tracking| Orchestrator

    Orchestrator -->|runs| DC
    DC -->|outputs to| Raw
    Orchestrator -->|runs| DP
    DP -->|reads from Raw<br>writes to| Int
    Orchestrator -->|runs| FE
    FE -->|reads from Int<br>writes to| Int
    Orchestrator -->|runs| DPr
    DPr -->|reads from Int<br>writes to| Int
    Orchestrator -->|runs| MB
    MB -->|reads from Int<br>writes to| Models
    Orchestrator -->|runs| ME
    ME -->|reads from Models<br>writes to| Metrics

    DVCIgnore -.->|ignore rules| Orchestrator
    DVCConf -.->|config| Orchestrator
    Orchestrator -.->|push/pull artifacts| DVCRemote
    Raw -.->|versioned artifacts| DVCRemote
    Int -.->|versioned artifacts| DVCRemote
    Models -.->|versioned artifacts| DVCRemote
    Metrics -.->|versioned artifacts| DVCRemote

    %% Styles
    classDef module fill:#bbdefb,stroke:#0d47a1;
    classDef data fill:#c8e6c9,stroke:#1b5e20,shape:cylinder;
    classDef orchestrator fill:#ffe0b2,stroke:#e65100,shape:parallelogram;
    classDef remote fill:#e0e0e0,stroke:#424242,shape:cylinder;
    classDef dvcmeta fill:#f3e5f5,stroke:#4a148c;
    classDef docs fill:#eeeeee,stroke:#616161;
```
