# tuananhne1110/mlops_for_yolov11 architecture

> Architecture diagram of the GitHub repository tuananhne1110/mlops_for_yolov11, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tuananhne1110/mlops_for_yolov11
- Repository: https://github.com/tuananhne1110/mlops_for_yolov11

## Overview

##### Main Components
- MLOps Infrastructure Services:
- Docker Services: Representing containerized deployment that brings up MLflow, PostgreSQL, and MinIO.
- MLflow: For experiment tracking, model registry, and logging metrics/artifacts.
- PostgreSQL: Storing MLflow backend data.
- MinIO: S3-compatible storage for datasets and artifacts.
- Data and Versioning Pipeline:
- DVC: For tracking different versions of the dataset.
- MinIO: As the final destination for versioned data and large files.
- Training Pipeline:
- Configuration Management: Utilizing YAML files (e.g., configs/training_config.yaml) for setting hyperparameters and model architecture.
- Model Training: Represented by src/training/train.py and supported utilities (like src/utils/config.py).
- Checkpointing & Logging: Captured via MLflow with scripted interactions (e.g., src/models/mlflow_model_wrapper.py and src/utils/mlflow_logger.py).
- Inference/Model Deployment:
- Inference Code under src/inference for running the trained models.
- Model Registry and version selection managed by MLflow.
- Supporting Scripts:
- Scripts under the scripts folder (for data management, service starting, dataset validation, and DB initialization) that automate various aspects of the workflow.

##### Relationships and Interactions
- Data Flow:
- You start with raw data that enters through the data pipeline.
- DVC processes and versions the data, which then flows into MinIO for storage.
- Training Process:
- The training pipeline fetches data and configuration, executes training (utilizing GPU resources), checkpoints and logs progress, and ultimately stores artifacts.
- MLflow is used to capture metrics, hyperparameters, and produced artifacts throughout the training process.
- MLOps Infrastructure:
- Inference & Deployment:
- Post-training, the model can be retrieved from the MLflow registry and deployed via inference service, with a flow showing how version control transitions into deployment.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Clearly divided directories for configuration (configs), core training and inference logic (src), and automation scripts (scripts).
- Containerization: Deployment of each service (MLflow, PostgreSQL, MinIO) within Docker containers as specified by docker-compose.yml.
- Modularity: The project splits various concerns like configuration, training, and utilities into distinct modules, allowing for easy updates and scaling.
- Automation & Reproducibility: The use of DVC for data versioning and Docker for replicable environments ensures consistency across experiments.

##### Relevant Technologies and Frameworks
- Python 3.8+ is the base language.
- MLflow for experiment tracking.
- DVC for data management.
- MinIO for object storage.
- PostgreSQL as the metadata store.
- Docker (and docker-compose) for container orchestration.
- Scripts indicate support for automation of various tasks (dataset management, service startup, etc.).

##### Key Inclusions
- Clear labels for:
- Each containerized service (MLflow, PostgreSQL, MinIO)
- Training, inference, and utility modules (src/training, src/inference, src/models, src/utils)
- Scripts handling the orchestration of the pipeline (scripts directory)
- How raw data moves into DVC and eventually into MinIO.
- The sequence of events in the training process (configuration → training → checkpointing → MLflow logging).
- Interaction between MLflow and its backend (PostgreSQL) as well as artifact storage (MinIO).
- For core functionality (e.g., training vs. inference).
- For data flow vs. control/configuration flow.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Pipeline"
        rawData["Raw Data"]:::data
        dvc["DVC Data Management"]:::data
    end
    rawData -->|"versioning"| dvc

    subgraph "MLOps Infrastructure"
        dockerCompose["Docker Compose Orchestration"]:::infrastructure
        mlflow["MLflow Experiment Tracker"]:::infrastructure
        postgresql["PostgreSQL Backend"]:::infrastructure
        minio["MinIO Storage"]:::infrastructure
    end
    dockerCompose -->|"orchestrates"| mlflow
    dockerCompose -->|"orchestrates"| postgresql
    dockerCompose -->|"orchestrates"| minio
    mlflow -->|"stores_data_in"| postgresql
    mlflow -->|"stores_artifacts_in"| minio

    subgraph "Training Pipeline"
        configuration["Configuration (YAML)"]:::training
        modelTraining["Model Training"]:::training
        utilityConfig["Utility Config"]:::training
        mlflowLogging["MLflow Logging & Wrapper"]:::training
    end
    configuration -->|"feeds"| modelTraining
    utilityConfig -->|"supports"| modelTraining
    modelTraining -->|"checkpoints_to"| mlflowLogging
    mlflowLogging -->|"logs_to"| mlflow
    dvc -->|"provides_data"| modelTraining

    subgraph "Model Deployment"
        inference["Inference Service"]:::inference
    end
    mlflow -->|"provides_model_to"| inference

    subgraph "Supporting Scripts"
        automation["Automation Scripts"]:::scripts
    end
    automation -->|"orchestrates"| dockerCompose

    %% Click Events
    click dockerCompose "https://github.com/tuananhne1110/mlops_for_yolov11/blob/main/docker-compose.yml"
    click mlflow "https://github.com/tuananhne1110/mlops_for_yolov11/blob/main/src/models/mlflow_model_wrapper.py"
    click postgresql "https://github.com/tuananhne1110/mlops_for_yolov11/blob/main/docker-compose.yml"
    click minio "https://github.com/tuananhne1110/mlops_for_yolov11/blob/main/docker-compose.yml"
    click dvc "https://github.com/tuananhne1110/mlops_for_yolov11/blob/main/scripts/init_dvc.py"
    click configuration "https://github.com/tuananhne1110/mlops_for_yolov11/blob/main/configs/training/training_config.yaml"
    click modelTraining "https://github.com/tuananhne1110/mlops_for_yolov11/blob/main/src/training/train.py"
    click utilityConfig "https://github.com/tuananhne1110/mlops_for_yolov11/blob/main/src/utils/config.py"
    click mlflowLogging "https://github.com/tuananhne1110/mlops_for_yolov11/blob/main/src/utils/mlflow_logger.py"
    click inference "https://github.com/tuananhne1110/mlops_for_yolov11/tree/main/src/inference"
    click automation "https://github.com/tuananhne1110/mlops_for_yolov11/blob/main/scripts/start_services.py"

    %% Styles
    classDef infrastructure fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef training fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef data fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef inference fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef scripts fill:#f8d7da,stroke:#721c24,stroke-width:2px;
```
