How sd031/mlops-pipeline works
Project Type and Purpose
• This is an end-to-end MLOps pipeline project. It ingests raw data, validates and transforms it, trains ML models, registers model artifacts, serves predictions via an API, and monitors model performance and data drift in production.
File-Structure Analysis
• Top-level directories:
• src/ – core application code split into api, data, models, monitoring, utils
• data/ – folders for raw, external, features, processed data
• models/ – artifacts, experiments, registry
• scripts/ & notebooks/ – orchestration scripts and exploratory notebooks
• docker/ & docker-compose.yml – container definitions for api, training, monitoring
• infrastructure/ – IaC stubs for Kubernetes and Terraform
• monitoring/ – Prometheus and Grafana configs/dashboards
• Config files (config/*.yaml), Makefile, requirements.txt, setup.py—indicate a containerized Python service with configurable pipelines and logging
Key Components
a. Data Ingestion & Storage
– Raw data sources (external) → ingestion service (src/data/ingestion.py) → raw data store
– Validation (src/data/validation.py)
– Feature engineering pipeline (scripts/run_pipeline.py or src/data) → features/processed storage
b. Model Training & Registry
– Training container (docker/Dockerfile.training) running src/models/train.py
– Model evaluation (src/models/evaluate.py)
– Artifact storage: models/artifacts & models/registry
c. Model Serving (API)
– API service container (docker/Dockerfile.api) launching src/api/app.py
– Prediction endpoint using src/models/predict.py and registry lookup
– Schemas (src/api/schemas.py)
– utils/logger.py & database.py for request logging and metadata persistence
d. Monitoring & Alerting
– Monitoring container (docker/Dockerfile.monitoring) running src/monitoring/drift.py & performance.py
– Metrics exported to Prometheus (monitoring/prometheus.yml)
– Grafana dashboards (monitoring/grafana/dashboards/)
e. Infrastructure & Orchestration
– docker-compose.yml unites api, training, monitoring services plus a database or message queue if present
– Kubernetes manifests and Terraform modules under infrastructure/ for production deployment
1. Data Source → Ingestion Service → Raw Data Store
2. Raw Data Store → Validation → Feature Store → Training Job
3. Training Job → Model Artifact → Model Registry
4. Model Registry → API Service → Client Applications
5. API Service & Training Job emit metrics → Prometheus
6. Prometheus → Grafana Dashboards → MLOps Team
7. IaC (Terraform/K8s) manages deployment of containers and storage
Architectural Patterns & Principles
• Microservices / Containerized Components: each stage (ingestion, training, serving, monitoring) runs in its own container
• Separation of Concerns: data, modeling, serving, and monitoring code live in dedicated modules
• Immutable Artifacts: model registry for reproducibility
• Infrastructure-as-Code: docker-compose for local, Terraform + Kubernetes for prod
• Observability: logging (config/logging.yaml), metrics (Prometheus), dashboards (Grafana)
Essential Elements Checklist
• Technology stack callouts: Python modules, Docker, Kubernetes, Terraform, Prometheus, Grafana
• IaC boundary: separate local Docker-Compose from production K8s/Terraform