How mminh007/mlops works
Project Type and Purpose
• This is a full-stack MLOps pipeline for training, tracking, and deploying a ResNet classification model.
• Key tools:
• Airflow for orchestration (flatform/airflow)
• MLflow for experiment tracking and model registry (flatform/mlflow)
• FastAPI as the model-serving backend (backend/)
• Streamlit as the user-facing frontend (frontend/)
• Containerization via Docker/Docker-Compose
File Structure Analysis
Top-level directories:
• backend/: FastAPI application (app code, API schema, v1 routes, controller)
• flatform/airflow/: DAG definitions, processing/train scripts, Airflow config, Docker-Compose for orchestration
• flatform/mlflow/: MLflow server config and Docker-Compose
• frontend/: Streamlit app
Patterns & components:
• “app/api/v1” suggests RESTful endpoint versioning
• Airflow uses a DAG to chain processing → training → logging
• MLflow deploys as a separate service for experiment tracking and model registry
• Dockerfiles/Docker-Compose for service isolation and deployment
Insights from README
• The pipeline builds & trains ResNet via Airflow, logs runs in MLflow
• After training, the model is served via FastAPI; Streamlit provides the UI
• Prometheus/Grafana mentioned for monitoring (future integration)
a. Main Components
1. Data Source (local files or external data store)
2. Airflow Scheduler & Workers (Orchestration)
3. DAG Tasks:
• Data Ingestion/Processing (processing.py)
• Model Training (train.py)
• Model Registration (register.py → MLflow)
4. MLflow Tracking Server & UI
5. Model Registry (MLflow)
6. FastAPI Backend (model serving endpoints)
7. Streamlit Frontend (user interaction, calls FastAPI)
8. Docker & Docker-Compose (containers for each service)
9. (Optional) Prometheus/Grafana for metrics
b. Relationships & Data Flows
• Airflow Driver (DAG) → Task Workers → MLflow (log metrics & artifacts)
• MLflow Model Registry → FastAPI loads latest model
• Frontend → HTTP calls → FastAPI → returns predictions
• Docker-Compose network connects Airflow, MLflow, FastAPI, Streamlit
c. Architectural Patterns & Principles
• Microservices style: each major function (orchestration, tracking, serving, UI) isolated in its own container
• DAG-based orchestration pattern for ML workflow reproducibility
• API versioning (v1) for backward-compatible endpoint evolution
• Separation of concerns: Training vs. Serving vs. UI
d. Key Technologies & Frameworks
• Apache Airflow
• MLflow Tracking & Model Registry
• Python (scripts, FastAPI, Streamlit)
• Docker / Docker-Compose
• Pydantic schemas (backend/app/api/schema)
• Git for version control
Final Advice
• Be exhaustive: include Docker-Compose networks and ports (e.g., 8080 for Airflow UI, 5000 for FastAPI, 8501 for Streamlit)
• File-structure cues: annotate that Airflow’s Python scripts live under flatform/airflow/src, FastAPI code under backend/app/api/v1
• Detail versioning: show v1 controller/routes in FastAPI
• Extensibility: note where Prometheus/Grafana will hook in for metrics