How rajinikanthvadla-ai/mlops-lab-workshop works
This repository is an end-to-end MLOps workshop for a churn prediction system, organized around a Kubernetes-hosted ML lifecycle rather than a single application. The architecture is split into five major runtime subsystems plus operational scaffolding.
1) Infrastructure and deployment layer
The entire stack runs in a local Kind Kubernetes cluster. The k8s/ directory contains the deployment manifests for each runtime component, and scripts/*.ps1 provides Windows-based lifecycle automation for creating the cluster, deploying services, port-forwarding, and tearing everything down.
Key infrastructure boundaries:
• k8s/namespaces/ isolates workloads into separate Kubernetes namespaces.
• k8s/postgres/, k8s/minio/, k8s/mlflow/, k8s/airflow/, k8s/redis/, k8s/model-serving/, and k8s/frontend/ represent independently deployable subsystems.
• The manifests suggest a classic platform split: orchestration, storage, tracking, serving, and UI are separated at the cluster level.
2) Data and storage layer
Two central data stores underpin the system:
• MinIO acts as the S3-compatible object store and functions as the data lake and artifact store.
- Raw input data lands in MinIO under a raw zone.
- Processed features are written back to MinIO under a processed zone.
- MLflow model artifacts are also stored in MinIO.
• PostgreSQL provides backend persistence.
- It is used by MLflow as its tracking/registry metadata store.
- It is also used by Airflow as its metadata database.
Redis is a separate online store used only by the serving layer, acting as a low-latency cache/feature store for inference-time lookups.
3) Workflow orchestration layer
Apache Airflow is the control plane for the ML pipeline. The dags/ directory contains the production DAGs:
• 01_data_ingestion.py
• 02_feature_engineering.py
• 03_model_training.py
• 04_model_evaluation.py
• 05_full_pipeline.py
There are also example DAGs for demo purposes.
The orchestration logic is staged:
• Data ingestion runs daily.
• Feature engineering runs daily.
• Model training runs weekly.
• Model evaluation runs daily.
• A full pipeline DAG coordinates the end-to-end sequence weekly.
Airflow is therefore the system boundary that connects external data acquisition, internal feature processing, model training, and model promotion decisions.
A notable repo-specific detail is the Git sync setup:
• github-dags-template/ and github-dags-repo/ indicate an alternate DAG distribution path where Airflow can sync DAGs from a Git-backed repository instead of relying only on local filesystem mounts.
• docs/GITHUB_SYNC_SETUP.md and the airflow-git-sync-configmap.yaml manifest show that DAG delivery is itself a managed part of the architecture.
4) ML development and registry layer
The core ML code lives under src/ and is split by concern:
• src/data/download_data.py
- Pulls the IBM Telco churn dataset from an external source.
- This is the system’s external data ingress point.
• src/features/feature_engineering.py
- Transforms raw customer records into model-ready features.
- Produces engineered outputs that are stored back into the data lake.
• src/training/train_model.py
- Trains multiple churn models.
- The README identifies the model families as logistic regression, random forest, and gradient boosting.
- Training outputs are registered into MLflow, which provides experiment tracking and model versioning.
This layer is the transition point between offline data processing and governed model management. MLflow is the key boundary service here: it captures metrics, artifacts, and model versions, while PostgreSQL and MinIO back its state.
5) Online serving layer
The runtime inference service lives in src/serving/ and is containerized independently with its own Dockerfile and requirements file.
Its internal modules are:
• app/main.py — FastAPI entrypoint and HTTP API surface
• app/predictor.py — model loading and prediction logic
• app/ab_router.py — A/B traffic routing and experiment assignment
The serving layer depends on:
• MLflow/model registry for deployed model selection
• MinIO for artifact retrieval, indirectly through MLflow-backed model storage
• Redis for online cache/feature lookup
• Kubernetes service exposure via k8s/model-serving/
The API is designed for both single and batch inference, plus explainability and health/metrics endpoints. The README lists:
• /predict
• /predict/batch
• /predict/explain
• /models
• /ab-stats
• /health
• /metrics
The A/B router is an important architectural boundary: it splits traffic between a champion and challenger model, with the stated default split being 80/20. That means the serving service is not just a stateless API; it also contains experiment-routing logic.
6) Frontend and observability-facing UX
src/frontend/ contains a React + Tailwind dashboard, deployed separately via Kubernetes manifests.
This UI consumes the model-serving API to show:
• real-time churn predictions
• model comparison
• A/B test visualization
• feature importance views
Architecturally, the frontend is a thin consumer of the serving API. It does not own ML logic or storage; it is a presentation layer for prediction and experiment status.
7) End-to-end data flow
The most important cross-system flow is:
1. External churn data is downloaded from IBM by the ingestion job.
2. Raw data is stored in MinIO.
3. Feature engineering reads the raw zone and writes processed features back to MinIO.
4. Training reads processed features and logs models/metrics/artifacts to MLflow.
5. MLflow persists metadata in PostgreSQL and artifacts in MinIO.
6. Evaluation compares registered models and updates the A/B decision state.
7. The serving layer loads approved models and uses Redis for online lookup/cache.
8. The frontend queries the FastAPI service to visualize predictions and model performance.
9. Airflow coordinates all of the above through scheduled DAGs.
8) Repo structure as architectural boundaries
The repository is intentionally split into layers that mirror the runtime architecture:
• src/ = reusable application code and inference/training logic
• dags/ = orchestration and pipeline control
• k8s/ = infrastructure-as-code for platform deployment
• scripts/ = operational automation for local cluster lifecycle
• docs/ = deployment and Git sync guidance
• github-dags-template/ and github-dags-repo/ = the Git-based DAG distribution pattern
9) What is most important for a diagram
If you draw the architecture, the key boxes and arrows should be:
• Kind Kubernetes cluster as the hosting boundary
• Airflow as the orchestrator
• MinIO as the shared storage plane for raw data, processed features, and artifacts
• PostgreSQL as the metadata backend for Airflow and MLflow
• MLflow as the experiment tracking and model registry service
• Redis as the online inference cache / feature store
• FastAPI serving as the model inference API with A/B routing
• React dashboard as the client of the serving API
• IBM Telco dataset as the external upstream data source
The most important dependency chain is:
external data source → MinIO raw zone → feature engineering → MinIO processed zone → training → MLflow registry/artifacts → serving/A-B routing → frontend/dashboard.
Main entry points: kind-cluster-config.yaml Kind cluster · main.py Serving API · mlflow-deployment.yaml MLflow · minio-deployment.yaml MinIO
Components 18
Platform
Kubernetes-hosted runtime and storage services
- Kind cluster · Local Kubernetes host for the workshop stack
kind-cluster-config.yaml - Namespaces · Namespace boundaries for deployable subsystems
k8s/namespaces/namespaces.yaml - MinIO · Shared data lake and artifact store
k8s/minio/minio-deployment.yaml - Postgres · Backing store for Airflow and MLflow metadata
k8s/postgres/postgres-deployment.yaml - MLflow · Experiment tracking and model registry service
k8s/mlflow/mlflow-deployment.yaml - Airflow · Pipeline control plane and scheduled DAG runner
k8s/airflow/airflow-webserver.yaml - Redis · Low-latency lookup store for inference time
k8s/redis/redis-deployment.yaml
Pipeline
Orchestration and ML lifecycle jobs
- Ingestion job · Downloads the IBM Telco churn dataset
src/data/download_data.py - Feature engineering · Builds model-ready features from raw customer data
src/features/feature_engineering.py - Training job · Trains churn models and logs runs
src/training/train_model.py - Production DAGs · Staged pipeline definitions for ingestion, features, training, and evaluation
dags/05_full_pipeline.py - Git DAG sync · Git-backed DAG delivery path for Airflow
k8s/airflow/airflow-git-sync-configmap.yaml - IBM Telco data · Upstream churn dataset used by ingestion
Applications
Training, serving, and UI code
- Serving API · Model inference service with routing and prediction endpoints
src/serving/app/main.py - A/B router · Champion/challenger routing logic for inference
src/serving/app/ab_router.py - Predictor · Model loading and prediction execution
src/serving/app/predictor.py - Dashboard · Client UI for predictions, model comparison, and A/B views
src/frontend/src/App.jsx
Ops
Cluster lifecycle and deployment automation
- Deploy scripts · Cluster creation, deployment, port-forwarding, and cleanup automation
scripts/deploy-all.ps1
Connections 26
- Deploy scripts → Kind cluster: create/deploy · Bootstraps and manages the local cluster
- Kind cluster → Namespaces: hosts · Cluster boundary for isolated workloads
- Kind cluster → Airflow: runs · Airflow deployed into Kubernetes
- Kind cluster → MLflow: runs · MLflow deployed into Kubernetes
- Kind cluster → MinIO: runs · Shared storage service in-cluster
- Kind cluster → Postgres: runs · Metadata database in-cluster
- Kind cluster → Redis: runs · Inference cache in-cluster
- Kind cluster → Serving API: runs · Model serving workload in-cluster
- Kind cluster → Dashboard: runs · Dashboard workload in-cluster
- Git DAG sync → Airflow: syncs · Alternative DAG delivery mechanism
- Airflow → Production DAGs: orchestrates · Schedules and coordinates pipeline stages
- Production DAGs → Ingestion job: executes · Ingestion stage definition
- Production DAGs → Feature engineering: executes · Feature engineering stage definition
- Production DAGs → Training job: executes · Training stage definition
- Ingestion job → IBM Telco data: pulls · Fetches the external churn dataset
- Ingestion job → MinIO: writes raw · Stores raw data in the lake
- Feature engineering → MinIO: reads/writes · Consumes raw data and persists processed features
- Training job → MinIO: reads features · Loads processed training data
- Training job → MLflow: logs · Stores metrics, artifacts, and model versions
- MLflow → Postgres: metadata · Tracks runs and registry state
- MLflow → MinIO: artifacts · Persists model artifacts and files
- Serving API → MLflow: loads model · Resolves approved model versions
- Serving API → Redis: queries · Uses online cache for inference lookups
- A/B router → Serving API: inside · Routing logic embedded in the serving service
- Predictor → Serving API: inside · Prediction engine embedded in the serving service
- Dashboard → Serving API: calls api · Consumes prediction and experiment endpoints