# mminh007/mlops architecture

> Architecture diagram of the GitHub repository mminh007/mlops, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mminh007/mlops
- Repository: https://github.com/mminh007/mlops

## Overview

##### Project Type and Purpose
- This is a full-stack MLOps pipeline for training, tracking, and deploying a ResNet classification model.
- Key tools:
- Airflow for orchestration (flatform/airflow)
- MLflow for experiment tracking and model registry (flatform/mlflow)
- FastAPI as the model-serving backend (backend/)
- Streamlit as the user-facing frontend (frontend/)
- Containerization via Docker/Docker-Compose

##### File Structure Analysis
Top-level directories:
- backend/: FastAPI application (app code, API schema, v1 routes, controller)
- flatform/airflow/: DAG definitions, processing/train scripts, Airflow config, Docker-Compose for orchestration
- flatform/mlflow/: MLflow server config and Docker-Compose
- frontend/: Streamlit app

Patterns & components:
- “app/api/v1” suggests RESTful endpoint versioning
- Airflow uses a DAG to chain processing → training → logging
- MLflow deploys as a separate service for experiment tracking and model registry
- Dockerfiles/Docker-Compose for service isolation and deployment

##### Insights from README
- The pipeline builds & trains ResNet via Airflow, logs runs in MLflow
- After training, the model is served via FastAPI; Streamlit provides the UI
- Prometheus/Grafana mentioned for monitoring (future integration)

a. Main Components
1. Data Source (local files or external data store)
2. Airflow Scheduler & Workers (Orchestration)
3. DAG Tasks:
- Data Ingestion/Processing (processing.py)
- Model Training (train.py)
- Model Registration (register.py → MLflow)
4. MLflow Tracking Server & UI
5. Model Registry (MLflow)
6. FastAPI Backend (model serving endpoints)
7. Streamlit Frontend (user interaction, calls FastAPI)
8. Docker & Docker-Compose (containers for each service)
9. (Optional) Prometheus/Grafana for metrics

b. Relationships & Data Flows
- Airflow Driver (DAG) → Task Workers → MLflow (log metrics & artifacts)
- MLflow Model Registry → FastAPI loads latest model
- Frontend → HTTP calls → FastAPI → returns predictions
- Docker-Compose network connects Airflow, MLflow, FastAPI, Streamlit

c. Architectural Patterns & Principles
- Microservices style: each major function (orchestration, tracking, serving, UI) isolated in its own container
- DAG-based orchestration pattern for ML workflow reproducibility
- API versioning (v1) for backward-compatible endpoint evolution
- Separation of concerns: Training vs. Serving vs. UI

d. Key Technologies & Frameworks
- Apache Airflow
- MLflow Tracking & Model Registry
- Python (scripts, FastAPI, Streamlit)
- Docker / Docker-Compose
- Pydantic schemas (backend/app/api/schema)
- Git for version control

##### Final Advice
- Be exhaustive: include Docker-Compose networks and ports (e.g., 8080 for Airflow UI, 5000 for FastAPI, 8501 for Streamlit)
- File-structure cues: annotate that Airflow’s Python scripts live under flatform/airflow/src, FastAPI code under backend/app/api/v1
- Detail versioning: show v1 controller/routes in FastAPI
- Extensibility: note where Prometheus/Grafana will hook in for metrics

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Source
    DS["Data Source"]:::data

    %% Orchestration Layer
    subgraph "Orchestration (Airflow)" 
        AO["Airflow Orchestration"]:::orchestration
        subgraph "DAG: flatform/airflow/dags/dag.py"
            P["Data Ingestion\nprocessing.py"]:::task
            T["Model Training\ntrain.py"]:::task
            R["Model Registration\nregister.py"]:::task
        end
        U["Utils Module\nsrc/utils/"]:::utility
        AO -->|triggers| P
        P -->|calls| T
        T -->|logs to| R
    end

    %% Tracking Layer
    subgraph "Tracking (MLflow)"
        MLFS["MLflow Server & UI"]:::tracking
        MLFR["Model Registry"]:::storage
        MLFS -->|store artifacts| MLFR
    end

    %% Serving Layer
    subgraph "Serving (FastAPI)"
        API["FastAPI v1 API"]:::serving
        SCHEMA["Schemas & Configs"]:::serving
        CTRL["Controllers"]:::serving
        API -->|uses| SCHEMA
        API -->|routes to| CTRL
    end

    %% UI Layer
    subgraph "UI (Streamlit)"
        UI["Streamlit UI"]:::ui
    end

    %% Compose Services
    subgraph "Docker-Compose Networks"
        DC1["Airflow Compose"]:::config
        DC2["MLflow Compose"]:::config
        DC3["Backend Dockerfile"]:::config
        DC4["Frontend Dockerfile"]:::config
    end

    %% Data Flows
    DS -->|feeds data| AO
    R -->|logs metrics| MLFS
    MLFR -->|serving model| API
    UI -->|HTTP requests| API

    %% Networking
    DC1 --- AO
    DC2 --- MLFS
    DC3 --- API
    DC4 --- UI

    %% Click Events
    click AO "https://github.com/mminh007/mlops/tree/main/flatform/airflow/"
    click P "https://github.com/mminh007/mlops/blob/main/flatform/airflow/src/processing.py"
    click T "https://github.com/mminh007/mlops/blob/main/flatform/airflow/src/train.py"
    click R "https://github.com/mminh007/mlops/blob/main/flatform/airflow/src/register.py"
    click U "https://github.com/mminh007/mlops/tree/main/flatform/airflow/src/utils/"
    click MLFS "https://github.com/mminh007/mlops/tree/main/flatform/mlflow/"
    click DC1 "https://github.com/mminh007/mlops/blob/main/flatform/airflow/docker-compose.yaml"
    click DC2 "https://github.com/mminh007/mlops/blob/main/flatform/mlflow/docker-compose.yaml"
    click API "https://github.com/mminh007/mlops/tree/main/backend/"
    click UI "https://github.com/mminh007/mlops/tree/main/frontend/"
    click DC3 "https://github.com/mminh007/mlops/tree/main/backend/dockerfile"
    click DC4 "https://github.com/mminh007/mlops/tree/main/frontend/dockerfile"
    click CTRL "https://github.com/mminh007/mlops/blob/main/backend/app/api/v1/controller/model.py"
    click UI "https://github.com/mminh007/mlops/blob/main/frontend/main.py"

    %% Styles
    classDef data fill:#cccccc,stroke:#333,stroke-width:1px
    classDef orchestration fill:#cce5ff,stroke:#007bff,stroke-width:1px
    classDef task fill:#99ccff,stroke:#0056b3,stroke-width:1px
    classDef utility fill:#b3d7ff,stroke:#0056b3,stroke-width:1px
    classDef tracking fill:#d4edda,stroke:#28a745,stroke-width:1px
    classDef storage fill:#c3e6cb,stroke:#28a745,stroke-width:1px
    classDef serving fill:#ffe5b4,stroke:#ff8c00,stroke-width:1px
    classDef ui fill:#e2d5f5,stroke:#6f42c1,stroke-width:1px
    classDef config fill:#f0f0f0,stroke:#999,stroke-width:1px,stroke-dasharray: 5 5
```
