How tuananhne1110/mlops_for_yolov11 works
Main Components
• MLOps Infrastructure Services:
• Docker Services: Representing containerized deployment that brings up MLflow, PostgreSQL, and MinIO.
• MLflow: For experiment tracking, model registry, and logging metrics/artifacts.
• PostgreSQL: Storing MLflow backend data.
• MinIO: S3-compatible storage for datasets and artifacts.
• Data and Versioning Pipeline:
• DVC: For tracking different versions of the dataset.
• MinIO: As the final destination for versioned data and large files.
• Training Pipeline:
• Configuration Management: Utilizing YAML files (e.g., configs/training_config.yaml) for setting hyperparameters and model architecture.
• Model Training: Represented by src/training/train.py and supported utilities (like src/utils/config.py).
• Checkpointing & Logging: Captured via MLflow with scripted interactions (e.g., src/models/mlflow_model_wrapper.py and src/utils/mlflow_logger.py).
• Inference/Model Deployment:
• Inference Code under src/inference for running the trained models.
• Model Registry and version selection managed by MLflow.
• Supporting Scripts:
• Scripts under the scripts folder (for data management, service starting, dataset validation, and DB initialization) that automate various aspects of the workflow.
Relationships and Interactions
• Data Flow:
• You start with raw data that enters through the data pipeline.
• DVC processes and versions the data, which then flows into MinIO for storage.
• Training Process:
• The training pipeline fetches data and configuration, executes training (utilizing GPU resources), checkpoints and logs progress, and ultimately stores artifacts.
• MLflow is used to capture metrics, hyperparameters, and produced artifacts throughout the training process.
• MLOps Infrastructure:
• Inference & Deployment:
• Post-training, the model can be retrieved from the MLflow registry and deployed via inference service, with a flow showing how version control transitions into deployment.
Architectural Patterns and Design Principles
• Separation of Concerns: Clearly divided directories for configuration (configs), core training and inference logic (src), and automation scripts (scripts).
• Containerization: Deployment of each service (MLflow, PostgreSQL, MinIO) within Docker containers as specified by docker-compose.yml.
• Modularity: The project splits various concerns like configuration, training, and utilities into distinct modules, allowing for easy updates and scaling.
• Automation & Reproducibility: The use of DVC for data versioning and Docker for replicable environments ensures consistency across experiments.
Relevant Technologies and Frameworks
• Python 3.8+ is the base language.
• MLflow for experiment tracking.
• DVC for data management.
• MinIO for object storage.
• PostgreSQL as the metadata store.
• Docker (and docker-compose) for container orchestration.
• Scripts indicate support for automation of various tasks (dataset management, service startup, etc.).
Key Inclusions
• Clear labels for:
• Each containerized service (MLflow, PostgreSQL, MinIO)
• Training, inference, and utility modules (src/training, src/inference, src/models, src/utils)
• Scripts handling the orchestration of the pipeline (scripts directory)
• How raw data moves into DVC and eventually into MinIO.
• The sequence of events in the training process (configuration → training → checkpointing → MLflow logging).
• Interaction between MLflow and its backend (PostgreSQL) as well as artifact storage (MinIO).
• For core functionality (e.g., training vs. inference).
• For data flow vs. control/configuration flow.