How mubahsirhassan/mlops-project works
Project Type and Purpose
• This is an end-to-end MLOps pipeline (maturity level 4) that covers data versioning, CI/CD, model serving, monitoring, drift detection, and continuous training.
• Core purpose: automate training, deployment, and monitoring of a ML model on AWS (ECR/EKS) with observability (Prometheus/Grafana) and drift monitoring (Streamlit).
Main Architectural Components
a. Source Control & Data Versioning
– GitHub repo (code, DVC metadata).
– DVC remote storage (e.g., S3) for datasets & model artifacts.
b. CI/CD Pipeline (GitHub Actions)
– main.yml workflow:
• Build Docker image
• Run pytest
• Push image to AWS ECR
• Trigger deployment to EKS.
c. Model Training & Experiment Tracking
– prediction_model module (pipeline.py, training_pipeline.py, processing/)
– MLflow tracking server (stores runs, parameters, metrics)
d. Model Serving
– FastAPI application (main.py & predict.py) packaged in Docker
– Deployed to AWS EKS (Kubernetes Deployment + Service via deployment.yml & service.yml)
– Exposes REST endpoints for real‐time and batch predictions
e. Monitoring & Observability
– FastAPI “/metrics” endpoint (Prometheus exporter)
– Prometheus scrapes metrics
f. Drift Monitoring App
– Streamlit app (drift_monitoring/app_v1.py)
– Connects to production data stream and historical data in DVC to detect data/target drift and quality issues
Relationships & Data Flows
• Data flow from raw data → DVC → preprocessing → training → trained model → DVC → MLflow
• CI/CD flow: Git push → GitHub Actions → Docker/ECR → Kubernetes
• Request flow: Client → FastAPI (EKS) → Model artifact
• Monitoring flow: FastAPI → Prometheus → Grafana
• Drift flow: Production data store → Streamlit app → Visual reports
Architectural Patterns & Principles
• Microservices (FastAPI service separated from drift App)
• GitOps/CICD (GitHub Actions triggers)
• Infrastructure as code (Kubernetes YAMLs, Dockerfile)
• DataOps/Data Versioning (DVC)
• Observability (Prometheus/Grafana)
• Experiment tracking (MLflow)
Technologies & Frameworks
• DVC (data + model versioning)
• GitHub Actions (CI/CD)
• Docker & AWS ECR (container registry)
• AWS EKS (Kubernetes cluster)
• FastAPI (model serving)
• MLflow (experiment tracking)
• Prometheus (metrics collection)
• Grafana (visualization)
• Streamlit (drift monitoring UI)
• PyTest (unit tests)