# pewpenguin/mlops-continuous-training-pipeline architecture

> Architecture diagram of the GitHub repository pewpenguin/mlops-continuous-training-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pewpenguin/mlops-continuous-training-pipeline
- Repository: https://github.com/pewpenguin/mlops-continuous-training-pipeline

## Overview

##### Project type and purpose
- This is an MLOps pipeline project that automates the full lifecycle of machine learning models: data processing, model training & evaluation, model performance monitoring, and deployment as a REST API.
- It integrates DVC for data/model versioning, MLflow for experiment tracking/registry, and uses GitHub Actions for CI/CD.

##### File structure
- Top-level orchestrator:
- cli.py, main.py, run_api.py, run_model_pipeline.py, run_monitoring.py
These scripts invoke the various pipeline stages and the API server.
- Core pipeline modules:
- pipelines/data_pipeline.py — data ingestion, cleaning, feature engineering
- pipelines/model_pipeline.py — model training, evaluation, registry operations
- pipelines/mlflow_utils.py — wrappers around MLflow tracking & registry
- Serving layer:
- api/app.py — Flask (or FastAPI) application bootstrap
- api/model_api.py — endpoint definitions to serve model predictions
- Monitoring layer:
- monitoring/data_drift.py — detects data drift with live/recorded data
- Configuration/examples:
- examples/*.json — config templates passed to CLI for each stage
- Infrastructure & DevOps:
- .github/workflows/ci_cd.yml — CI/CD pipeline definitions
- setup.py, requirements.txt — package and dependency management
- Tests
- tests/test_pipelines.py — unit/integration tests for pipeline components

##### Extract extra architectural clues from README
- Features list confirms use of DVC, MLflow, REST API serving, drift monitoring, CI/CD.
- Usage examples show four commands driven by the CLI: process-data, train-model, monitor-model, serve-model.

##### Main components
a. CLI Orchestrator
- Single entry point (cli.py) invoked by users or CI
- Reads JSON configs → triggers one of the pipelines or the API.
b. Data Pipeline
- Source data (local or DVC-tracked) → data_pipeline → feature store (artifact directory / DVC remote)
c. Model Pipeline
- Features → model_pipeline → trained model artifact → model registry (MLflow) and DVC remote
- Experiment logs go to MLflow Tracking Server.
d. Monitoring Pipeline
- Production data / model predictions → data_drift.py → monitoring reports and alerts
- Optionally writes drift metrics back into MLflow or a dashboard.
e. Model Serving API
- api/app.py + api/model_api.py → loads model from MLflow Registry or DVC
- Exposes REST endpoints for inference requests.
f. External & Infrastructure Services
- DVC Remote (e.g., S3) for versioned data & models
- MLflow Server (Tracking UI, Model Registry)
- GitHub Actions for CI/CD (tests, builds, deployments)
g. Tests & CI/CD
- Tests invoked by CI — ensures data pipeline, model pipeline, API endpoints pass expected behavior.

##### Relationships and interactions
- API → Model Store (MLflow/DVC) to load artifacts
- Monitoring → API logs or live traffic
- CI/CD workflow monitors repo → runs tests → optionally deploys updated pipelines/API

##### Architectural patterns & principles
- Modular pipelines: each stage (data, model, monitoring) is a self-contained module
- Configuration-driven: JSON configs decouple code from environment specifics
- Version control integration: DVC for data/models, Git for code
- Experiment tracking: MLflow
- Clear separation of concerns: CLI orchestrator vs. library code vs. API layer

##### Technologies, frameworks, libraries
- Python, Click or argparse (CLI), Flask/FastAPI (API), DVC, MLflow, pytest, GitHub Actions, S3 (or other DVC remote)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        MainEntrypoint["Main Entrypoint"]:::cli
        CLI["CLI Orchestrator"]:::cli
        MainEntrypoint -->|"invokes"| CLI

        RunData["Data Pipeline Launcher"]:::cli
        RunModel["Model Pipeline Launcher"]:::cli
        RunMonitor["Monitoring Pipeline Launcher"]:::cli
        RunAPI["API Server Launcher"]:::cli

        CLI -->|"process-data config.json"| RunData
        CLI -->|"train-model config.json"| RunModel
        CLI -->|"monitor-model config.json"| RunMonitor
        CLI -->|"serve-model config.json"| RunAPI
    end

    %% Pipelines
    subgraph "Pipelines" 
        direction TB
        DataPipeline["Data Pipeline"]:::pipeline
        ModelPipeline["Model Pipeline"]:::pipeline
        MonitoringPipeline["Monitoring Pipeline"]:::pipeline

        RunData -->|"runs"| DataPipeline
        RunModel -->|"runs"| ModelPipeline
        RunMonitor -->|"runs"| MonitoringPipeline
    end

    %% Serving Layer
    subgraph "Serving Layer"
        direction TB
        APIApp["Model Serving API"]:::api
        RunAPI -->|"launches"| APIApp
    end

    %% External Services
    subgraph "External Services"
        direction TB
        DVC[(DVC Remote)]:::external
        MLflow[(MLflow Server)]:::external
        Clients{"Clients"}:::external

        DataPipeline -->|"data artifacts"| DVC
        ModelPipeline -->|"model artifacts"| DVC
        ModelPipeline -->|"experiment logs"| MLflow
        MonitoringPipeline -->|"drift metrics"| MLflow
        APIApp -->|"loads model"| MLflow
        APIApp -->|"fetches data/models"| DVC
        Clients -->|"HTTP requests"| APIApp
    end

    %% CI/CD
    subgraph "CI/CD"
        direction TB
        GHActions["GitHub Actions"]:::ci
        Tests["Tests (pipelines & API)"]:::ci

        GHActions -->|"runs"| Tests
        GHActions -->|"deploys"| RunData
        GHActions -->|"deploys"| RunModel
        GHActions -->|"deploys"| APIApp
    end

    %% Click Events
    click CLI "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/cli.py"
    click MainEntrypoint "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/main.py"
    click RunAPI "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/run_api.py"
    click RunModel "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/run_model_pipeline.py"
    click RunData "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/run_model_pipeline.py"
    click RunMonitor "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/run_monitoring.py"
    click DataPipeline "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/pipelines/data_pipeline.py"
    click ModelPipeline "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/pipelines/model_pipeline.py"
    click MonitoringPipeline "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/monitoring/data_drift.py"
    click APIApp "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/api/app.py"
    click MLflowUtils "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/pipelines/mlflow_utils.py"
    click MLflowUtils "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/pipelines/mlflow_utils.py"
    click APIEndpoints "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/api/model_api.py"
    click APIEndpoints "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/api/model_api.py"
    click ConfigExamples "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/examples/data_config.json"
    click ConfigExamples "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/examples/model_config.json"
    click ConfigExamples "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/examples/monitor_config.json"
    click ConfigExamples "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/examples/api_config.json"
    click GHActions "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/.github/workflows/ci_cd.yml"
    click Tests "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/tests/test_pipelines.py"
    click Requirements "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/requirements.txt"
    click Setup "https://github.com/pewpenguin/mlops-continuous-training-pipeline/blob/main/setup.py"

    %% Styles
    classDef pipeline fill:#E0F7FA,stroke:#0288D1,stroke-width:1px
    classDef api fill:#E8F5E9,stroke:#388E3C,stroke-width:1px
    classDef external fill:#FFF3E0,stroke:#F57C00,stroke-width:1px
    classDef ci fill:#FCE4EC,stroke:#C2185B,stroke-width:1px
    classDef cli fill:#E0E0E0,stroke:#757575,stroke-width:1px
```
