How pewpenguin/mlops-continuous-training-pipeline works
Project type and purpose
• This is an MLOps pipeline project that automates the full lifecycle of machine learning models: data processing, model training & evaluation, model performance monitoring, and deployment as a REST API.
• It integrates DVC for data/model versioning, MLflow for experiment tracking/registry, and uses GitHub Actions for CI/CD.
File structure
• Top-level orchestrator:
• cli.py, main.py, run_api.py, run_model_pipeline.py, run_monitoring.py
These scripts invoke the various pipeline stages and the API server.
• Core pipeline modules:
• pipelines/data_pipeline.py — data ingestion, cleaning, feature engineering
• pipelines/model_pipeline.py — model training, evaluation, registry operations
• pipelines/mlflow_utils.py — wrappers around MLflow tracking & registry
• Serving layer:
• api/app.py — Flask (or FastAPI) application bootstrap
• api/model_api.py — endpoint definitions to serve model predictions
• Monitoring layer:
• monitoring/data_drift.py — detects data drift with live/recorded data
• Configuration/examples:
• examples/*.json — config templates passed to CLI for each stage
• Infrastructure & DevOps:
• .github/workflows/ci_cd.yml — CI/CD pipeline definitions
• setup.py, requirements.txt — package and dependency management
• Tests
• tests/test_pipelines.py — unit/integration tests for pipeline components
Extract extra architectural clues from README
• Features list confirms use of DVC, MLflow, REST API serving, drift monitoring, CI/CD.
• Usage examples show four commands driven by the CLI: process-data, train-model, monitor-model, serve-model.
Main components
a. CLI Orchestrator
• Single entry point (cli.py) invoked by users or CI
• Reads JSON configs → triggers one of the pipelines or the API.
b. Data Pipeline
• Source data (local or DVC-tracked) → data_pipeline → feature store (artifact directory / DVC remote)
c. Model Pipeline
• Features → model_pipeline → trained model artifact → model registry (MLflow) and DVC remote
• Experiment logs go to MLflow Tracking Server.
d. Monitoring Pipeline
• Production data / model predictions → data_drift.py → monitoring reports and alerts
• Optionally writes drift metrics back into MLflow or a dashboard.
e. Model Serving API
• api/app.py + api/model_api.py → loads model from MLflow Registry or DVC
• Exposes REST endpoints for inference requests.
f. External & Infrastructure Services
• DVC Remote (e.g., S3) for versioned data & models
• MLflow Server (Tracking UI, Model Registry)
• GitHub Actions for CI/CD (tests, builds, deployments)
g. Tests & CI/CD
• Tests invoked by CI — ensures data pipeline, model pipeline, API endpoints pass expected behavior.
Relationships and interactions
• API → Model Store (MLflow/DVC) to load artifacts
• Monitoring → API logs or live traffic
• CI/CD workflow monitors repo → runs tests → optionally deploys updated pipelines/API
Architectural patterns & principles
• Modular pipelines: each stage (data, model, monitoring) is a self-contained module
• Configuration-driven: JSON configs decouple code from environment specifics
• Version control integration: DVC for data/models, Git for code
• Experiment tracking: MLflow
• Clear separation of concerns: CLI orchestrator vs. library code vs. API layer
Technologies, frameworks, libraries
• Python, Click or argparse (CLI), Flask/FastAPI (API), DVC, MLflow, pytest, GitHub Actions, S3 (or other DVC remote)