How mohammmedab/mlops-pipeline-bike-trip-duration works
Main Components
a) Data Layer:
• AWS S3 Bucket (Training Data Storage)
• PostgreSQL Database (MLflow Backend & Monitoring)
• Data flow paths for both training and prediction data
b) Training & Experimentation Layer:
• MLflow Server
• Experiment Tracking
• Model Registry
• Prefect Orchestration
• Training Flow
• Batch Metrics Calculation Flow
c) Serving Layer:
• Flask API
• Docker Container
• Gunicorn Server
d) Monitoring Layer:
• Evidently (Metrics Calculation)
• Grafana Dashboards
• PostgreSQL Monitoring Database
Component Relationships
• S3 → Prefect (Training data flow)
• Prefect → MLflow (Model training and registration)
• MLflow → Flask API (Model serving)
• Flask API → PostgreSQL (Prediction logging)
• PostgreSQL → Evidently (Metrics calculation)
• Evidently → Monitoring DB → Grafana (Monitoring flow)
Infrastructure Layer
AWS services as the underlying infrastructure:
• EC2 Instance (hosting MLflow, Grafana, Monitoring DB)
• RDS PostgreSQL
• S3 Bucket
Specific Design Elements
Use the following visual elements:
• Blue: AWS Services
• Green: Data Storage
• Orange: Processing Services
• Purple: Monitoring Components
• Rectangles: Services
Important Details
• Prefect flows (orchestration/batch_metrics_calculation.py and orchestration/train.py)
• Web service endpoint (web/predict.py)
• Database connections (database/sql.py)
• Monitoring components (monitoring/dummy_metrics.py)