# sd031/mlops-pipeline architecture

> Architecture diagram of the GitHub repository sd031/mlops-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sd031/mlops-pipeline
- Repository: https://github.com/sd031/mlops-pipeline

## Overview

##### Project Type and Purpose
- This is an end-to-end MLOps pipeline project. It ingests raw data, validates and transforms it, trains ML models, registers model artifacts, serves predictions via an API, and monitors model performance and data drift in production.

##### File-Structure Analysis
- Top-level directories:
- src/ – core application code split into api, data, models, monitoring, utils
- data/ – folders for raw, external, features, processed data
- models/ – artifacts, experiments, registry
- scripts/ & notebooks/ – orchestration scripts and exploratory notebooks
- docker/ & docker-compose.yml – container definitions for api, training, monitoring
- infrastructure/ – IaC stubs for Kubernetes and Terraform
- monitoring/ – Prometheus and Grafana configs/dashboards
- Config files (config/*.yaml), Makefile, requirements.txt, setup.py—indicate a containerized Python service with configurable pipelines and logging

##### Key Components
a. Data Ingestion & Storage
– Raw data sources (external) → ingestion service (src/data/ingestion.py) → raw data store
– Validation (src/data/validation.py)
– Feature engineering pipeline (scripts/run_pipeline.py or src/data) → features/processed storage
b. Model Training & Registry
– Training container (docker/Dockerfile.training) running src/models/train.py
– Model evaluation (src/models/evaluate.py)
– Artifact storage: models/artifacts & models/registry
c. Model Serving (API)
– API service container (docker/Dockerfile.api) launching src/api/app.py
– Prediction endpoint using src/models/predict.py and registry lookup
– Schemas (src/api/schemas.py)
– utils/logger.py & database.py for request logging and metadata persistence
d. Monitoring & Alerting
– Monitoring container (docker/Dockerfile.monitoring) running src/monitoring/drift.py & performance.py
– Metrics exported to Prometheus (monitoring/prometheus.yml)
– Grafana dashboards (monitoring/grafana/dashboards/)
e. Infrastructure & Orchestration
– docker-compose.yml unites api, training, monitoring services plus a database or message queue if present
– Kubernetes manifests and Terraform modules under infrastructure/ for production deployment

1. Data Source → Ingestion Service → Raw Data Store
2. Raw Data Store → Validation → Feature Store → Training Job
3. Training Job → Model Artifact → Model Registry
4. Model Registry → API Service → Client Applications
5. API Service & Training Job emit metrics → Prometheus
6. Prometheus → Grafana Dashboards → MLOps Team
7. IaC (Terraform/K8s) manages deployment of containers and storage

##### Architectural Patterns & Principles
- Microservices / Containerized Components: each stage (ingestion, training, serving, monitoring) runs in its own container
- Separation of Concerns: data, modeling, serving, and monitoring code live in dedicated modules
- Immutable Artifacts: model registry for reproducibility
- Infrastructure-as-Code: docker-compose for local, Terraform + Kubernetes for prod
- Observability: logging (config/logging.yaml), metrics (Prometheus), dashboards (Grafana)

##### Essential Elements Checklist
- Technology stack callouts: Python modules, Docker, Kubernetes, Terraform, Prometheus, Grafana
- IaC boundary: separate local Docker-Compose from production K8s/Terraform

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        External["External Data"]:::data
        Ingestion["Data Ingestion Service"]:::data
        RawStore["Raw Data Store"]:::data
        Validation["Data Validation Module"]:::data
        FeatureStore["Feature Store"]:::data
    end

    External -->|"raw data"| Ingestion
    click Ingestion "https://github.com/sd031/mlops-pipeline/blob/main/src/data/ingestion.py"
    Ingestion -->|"stores raw"| RawStore
    RawStore -->|"validate data"| Validation
    click Validation "https://github.com/sd031/mlops-pipeline/blob/main/src/data/validation.py"
    Validation -->|"features"| FeatureStore

    %% Compute / ML Layer
    subgraph "Compute / ML Layer"
        Pipeline["Feature Engineering Pipeline"]:::compute
        Training["Model Training Job"]:::compute
        Evaluation["Model Evaluation"]:::compute
        ArtifactStore["Model Artifacts"]:::compute
        Registry["Model Registry"]:::compute
    end

    FeatureStore -->|"run pipeline"| Pipeline
    click Pipeline "https://github.com/sd031/mlops-pipeline/blob/main/scripts/run_pipeline.py"
    Pipeline -->|"train model"| Training
    click Training "https://github.com/sd031/mlops-pipeline/blob/main/src/models/train.py"
    Training -->|"evaluate"| Evaluation
    click Evaluation "https://github.com/sd031/mlops-pipeline/blob/main/src/models/evaluate.py"
    Evaluation -->|"store artifact"| ArtifactStore
    ArtifactStore -->|"register model"| Registry

    %% Serving Layer
    subgraph "Serving Layer"
        APIService["API Service"]:::serving
        Prediction["Prediction Logic"]:::serving
        Schemas["API Schemas"]:::serving
        UtilsLogger["Logging Utils"]:::serving
        UtilsDB["Database Utils"]:::serving
        Client["Client Applications"]:::external
    end

    Registry -->|"fetch model"| APIService
    click APIService "https://github.com/sd031/mlops-pipeline/blob/main/src/api/app.py"
    APIService -->|"predict"| Prediction
    click Prediction "https://github.com/sd031/mlops-pipeline/blob/main/src/models/predict.py"
    APIService -->|"validate/request"| Schemas
    click Schemas "https://github.com/sd031/mlops-pipeline/blob/main/src/api/schemas.py"
    APIService -->|"log request"| UtilsLogger
    click UtilsLogger "https://github.com/sd031/mlops-pipeline/blob/main/src/utils/logger.py"
    APIService -->|"persist metadata"| UtilsDB
    click UtilsDB "https://github.com/sd031/mlops-pipeline/blob/main/src/utils/database.py"
    Client -->|"request"| APIService

    %% Monitoring / Observability
    subgraph "Monitoring / Observability"
        Drift["Data Drift Monitoring"]:::monitor
        Performance["Performance Monitoring"]:::monitor
        Prometheus["Prometheus"]:::monitor
        GrafanaDash["Grafana Dashboards"]:::monitor
        GrafanaDS["Grafana Datasources"]:::monitor
    end

    Training -->|"metrics"| Prometheus
    APIService -->|"metrics"| Prometheus
    click Prometheus "https://github.com/sd031/mlops-pipeline/blob/main/monitoring/prometheus.yml"
    Prometheus --> GrafanaDash
    click GrafanaDash "https://github.com/sd031/mlops-pipeline/tree/main/monitoring/grafana/dashboards/"
    Prometheus --> GrafanaDS
    click GrafanaDS "https://github.com/sd031/mlops-pipeline/blob/main/monitoring/grafana/datasources/prometheus.yml"
    GrafanaDash -->|"alerts"| Drift
    click Drift "https://github.com/sd031/mlops-pipeline/blob/main/src/monitoring/drift.py"
    GrafanaDash -->|"alerts"| Performance
    click Performance "https://github.com/sd031/mlops-pipeline/blob/main/src/monitoring/performance.py"

    %% Infrastructure / Orchestration
    subgraph "Infrastructure / Orchestration"
        DockerCompose["Docker Compose"]:::infra
        TrainContainer["Training Container"]:::infra
        APIContainer["API Container"]:::infra
        MonitorContainer["Monitoring Container"]:::infra
        K8s["Kubernetes Manifests"]:::infra
        Terraform["Terraform Modules"]:::infra
    end

    DockerCompose --> Ingestion
    DockerCompose --> Training
    DockerCompose --> APIService
    DockerCompose --> Drift
    click DockerCompose "https://github.com/sd031/mlops-pipeline/blob/main/docker-compose.yml"
    click TrainContainer "https://github.com/sd031/mlops-pipeline/blob/main/docker/Dockerfile.training"
    click APIContainer "https://github.com/sd031/mlops-pipeline/blob/main/docker/Dockerfile.api"
    click MonitorContainer "https://github.com/sd031/mlops-pipeline/blob/main/docker/Dockerfile.monitoring"
    DockerCompose --> TrainContainer
    DockerCompose --> APIContainer
    DockerCompose --> MonitorContainer
    K8s --> DockerCompose
    click K8s "https://github.com/sd031/mlops-pipeline/tree/main/infrastructure/kubernetes/"
    Terraform --> K8s
    click Terraform "https://github.com/sd031/mlops-pipeline/tree/main/infrastructure/terraform/"

    %% Styles
    classDef data fill:#bbdefb,stroke:#0d47a1
    classDef compute fill:#c8e6c9,stroke:#1b5e20
    classDef serving fill:#ffcc80,stroke:#e65100
    classDef monitor fill:#e1bee7,stroke:#6a1b9a
    classDef infra fill:#eceff1,stroke:#37474f
    classDef external fill:#ffffff,stroke:#000000
```
