# rajinikanthvadla-ai/mlops-lab-workshop architecture

> Architecture diagram of the GitHub repository rajinikanthvadla-ai/mlops-lab-workshop, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rajinikanthvadla-ai/mlops-lab-workshop
- Repository: https://github.com/rajinikanthvadla-ai/mlops-lab-workshop

## Overview

This repository is an end-to-end MLOps workshop for a churn prediction system, organized around a Kubernetes-hosted ML lifecycle rather than a single application. The architecture is split into five major runtime subsystems plus operational scaffolding.

##### 1) Infrastructure and deployment layer
The entire stack runs in a local Kind Kubernetes cluster. The `k8s/` directory contains the deployment manifests for each runtime component, and `scripts/*.ps1` provides Windows-based lifecycle automation for creating the cluster, deploying services, port-forwarding, and tearing everything down.

Key infrastructure boundaries:
- `k8s/namespaces/` isolates workloads into separate Kubernetes namespaces.
- `k8s/postgres/`, `k8s/minio/`, `k8s/mlflow/`, `k8s/airflow/`, `k8s/redis/`, `k8s/model-serving/`, and `k8s/frontend/` represent independently deployable subsystems.
- The manifests suggest a classic platform split: orchestration, storage, tracking, serving, and UI are separated at the cluster level.

##### 2) Data and storage layer
Two central data stores underpin the system:

- MinIO acts as the S3-compatible object store and functions as the data lake and artifact store.
  - Raw input data lands in MinIO under a raw zone.
  - Processed features are written back to MinIO under a processed zone.
  - MLflow model artifacts are also stored in MinIO.

- PostgreSQL provides backend persistence.
  - It is used by MLflow as its tracking/registry metadata store.
  - It is also used by Airflow as its metadata database.

Redis is a separate online store used only by the serving layer, acting as a low-latency cache/feature store for inference-time lookups.

##### 3) Workflow orchestration layer
Apache Airflow is the control plane for the ML pipeline. The `dags/` directory contains the production DAGs:

- `01_data_ingestion.py`
- `02_feature_engineering.py`
- `03_model_training.py`
- `04_model_evaluation.py`
- `05_full_pipeline.py`

There are also example DAGs for demo purposes.

The orchestration logic is staged:
- Data ingestion runs daily.
- Feature engineering runs daily.
- Model training runs weekly.
- Model evaluation runs daily.
- A full pipeline DAG coordinates the end-to-end sequence weekly.

Airflow is therefore the system boundary that connects external data acquisition, internal feature processing, model training, and model promotion decisions.

A notable repo-specific detail is the Git sync setup:
- `github-dags-template/` and `github-dags-repo/` indicate an alternate DAG distribution path where Airflow can sync DAGs from a Git-backed repository instead of relying only on local filesystem mounts.
- `docs/GITHUB_SYNC_SETUP.md` and the `airflow-git-sync-configmap.yaml` manifest show that DAG delivery is itself a managed part of the architecture.

##### 4) ML development and registry layer
The core ML code lives under `src/` and is split by concern:

- `src/data/download_data.py`
  - Pulls the IBM Telco churn dataset from an external source.
  - This is the system’s external data ingress point.

- `src/features/feature_engineering.py`
  - Transforms raw customer records into model-ready features.
  - Produces engineered outputs that are stored back into the data lake.

- `src/training/train_model.py`
  - Trains multiple churn models.
  - The README identifies the model families as logistic regression, random forest, and gradient boosting.
  - Training outputs are registered into MLflow, which provides experiment tracking and model versioning.

This layer is the transition point between offline data processing and governed model management. MLflow is the key boundary service here: it captures metrics, artifacts, and model versions, while PostgreSQL and MinIO back its state.

##### 5) Online serving layer
The runtime inference service lives in `src/serving/` and is containerized independently with its own Dockerfile and requirements file.

Its internal modules are:
- `app/main.py` — FastAPI entrypoint and HTTP API surface
- `app/predictor.py` — model loading and prediction logic
- `app/ab_router.py` — A/B traffic routing and experiment assignment

The serving layer depends on:
- MLflow/model registry for deployed model selection
- MinIO for artifact retrieval, indirectly through MLflow-backed model storage
- Redis for online cache/feature lookup
- Kubernetes service exposure via `k8s/model-serving/`

The API is designed for both single and batch inference, plus explainability and health/metrics endpoints. The README lists:
- `/predict`
- `/predict/batch`
- `/predict/explain`
- `/models`
- `/ab-stats`
- `/health`
- `/metrics`

The A/B router is an important architectural boundary: it splits traffic between a champion and challenger model, with the stated default split being 80/20. That means the serving service is not just a stateless API; it also contains experiment-routing logic.

##### 6) Frontend and observability-facing UX
`src/frontend/` contains a React + Tailwind dashboard, deployed separately via Kubernetes manifests.

This UI consumes the model-serving API to show:
- real-time churn predictions
- model comparison
- A/B test visualization
- feature importance views

Architecturally, the frontend is a thin consumer of the serving API. It does not own ML logic or storage; it is a presentation layer for prediction and experiment status.

##### 7) End-to-end data flow
The most important cross-system flow is:

1. External churn data is downloaded from IBM by the ingestion job.
2. Raw data is stored in MinIO.
3. Feature engineering reads the raw zone and writes processed features back to MinIO.
4. Training reads processed features and logs models/metrics/artifacts to MLflow.
5. MLflow persists metadata in PostgreSQL and artifacts in MinIO.
6. Evaluation compares registered models and updates the A/B decision state.
7. The serving layer loads approved models and uses Redis for online lookup/cache.
8. The frontend queries the FastAPI service to visualize predictions and model performance.
9. Airflow coordinates all of the above through scheduled DAGs.

##### 8) Repo structure as architectural boundaries
The repository is intentionally split into layers that mirror the runtime architecture:

- `src/` = reusable application code and inference/training logic
- `dags/` = orchestration and pipeline control
- `k8s/` = infrastructure-as-code for platform deployment
- `scripts/` = operational automation for local cluster lifecycle
- `docs/` = deployment and Git sync guidance
- `github-dags-template/` and `github-dags-repo/` = the Git-based DAG distribution pattern

##### 9) What is most important for a diagram
If you draw the architecture, the key boxes and arrows should be:

- Kind Kubernetes cluster as the hosting boundary
- Airflow as the orchestrator
- MinIO as the shared storage plane for raw data, processed features, and artifacts
- PostgreSQL as the metadata backend for Airflow and MLflow
- MLflow as the experiment tracking and model registry service
- Redis as the online inference cache / feature store
- FastAPI serving as the model inference API with A/B routing
- React dashboard as the client of the serving API
- IBM Telco dataset as the external upstream data source

The most important dependency chain is:
external data source → MinIO raw zone → feature engineering → MinIO processed zone → training → MLflow registry/artifacts → serving/A-B routing → frontend/dashboard.

Main entry points: [`kind-cluster-config.yaml`](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/kind-cluster-config.yaml) (Kind cluster), [`src/serving/app/main.py`](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/src/serving/app/main.py) (Serving API), [`k8s/mlflow/mlflow-deployment.yaml`](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/k8s/mlflow/mlflow-deployment.yaml) (MLflow), [`k8s/minio/minio-deployment.yaml`](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/k8s/minio/minio-deployment.yaml) (MinIO)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph "Platform"
  kind{{"Kind cluster<br/>kubernetes runtime"}}
  ns["Namespaces<br/>cluster isolation<br/>[namespaces.yaml]"]
  minio[("MinIO<br/>object storage")]
  postgres[("Postgres<br/>metadata db")]
  mlflow["MLflow<br/>tracking/registry"]
  airflow["Airflow<br/>orchestrator"]
  redis[("Redis<br/>online cache")]
end

subgraph "Pipeline"
  download["Ingestion job<br/>data pull<br/>[download_data.py]"]
  featureeng["Feature engineering<br/>transform job"]
  train["Training job<br/>model training<br/>[train_model.py]"]
  dags["Production DAGs<br/>airflow dags"]
  gitdags["Git DAG sync<br/>dag distribution"]
  dataupstream(("IBM Telco data<br/>external source"))
end

subgraph "Applications"
  serving["Serving API<br/>fastapi inference<br/>[main.py]"]
  abrouter{{"A/B router<br/>traffic split<br/>[ab_router.py]"}}
  predictor["Predictor<br/>model runtime<br/>[predictor.py]"]
  frontend["Dashboard<br/>react ui<br/>[App.jsx]"]
end

subgraph "Ops"
  scripts["Deploy scripts<br/>powershell ops<br/>[deploy-all.ps1]"]
end

scripts -->|"create/deploy"| kind
kind -->|"hosts"| ns
kind -->|"runs"| airflow
kind -->|"runs"| mlflow
kind -->|"runs"| minio
kind -->|"runs"| postgres
kind -->|"runs"| redis
kind -->|"runs"| serving
kind -->|"runs"| frontend
gitdags -->|"syncs"| airflow
airflow -->|"orchestrates"| dags
dags -->|"executes"| download
dags -->|"executes"| featureeng
dags -->|"executes"| train
download -->|"pulls"| dataupstream
download -->|"writes raw"| minio
featureeng -->|"reads/writes"| minio
train -->|"reads features"| minio
train -->|"logs"| mlflow
mlflow -->|"metadata"| postgres
mlflow -->|"artifacts"| minio
serving -->|"loads model"| mlflow
serving -->|"queries"| redis
abrouter -->|"inside"| serving
predictor -->|"inside"| serving
frontend -->|"calls api"| serving

click kind "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/kind-cluster-config.yaml"
click ns "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/k8s/namespaces/namespaces.yaml"
click minio "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/k8s/minio/minio-deployment.yaml"
click postgres "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/k8s/postgres/postgres-deployment.yaml"
click mlflow "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/k8s/mlflow/mlflow-deployment.yaml"
click airflow "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/k8s/airflow/airflow-webserver.yaml"
click redis "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/k8s/redis/redis-deployment.yaml"
click serving "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/src/serving/app/main.py"
click abrouter "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/src/serving/app/ab_router.py"
click predictor "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/src/serving/app/predictor.py"
click frontend "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/src/frontend/src/App.jsx"
click download "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/src/data/download_data.py"
click featureeng "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/src/features/feature_engineering.py"
click train "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/src/training/train_model.py"
click dags "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/dags/05_full_pipeline.py"
click gitdags "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/k8s/airflow/airflow-git-sync-configmap.yaml"
click scripts "https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/main/scripts/deploy-all.ps1"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class kind,ns,minio,postgres,mlflow,airflow,redis toneBlue
class download,featureeng,train,dags,gitdags,dataupstream toneAmber
class serving,abrouter,predictor,frontend toneMint
class scripts toneRose
```

## Components

### Platform

Kubernetes-hosted runtime and storage services

- [Kind cluster](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/kind-cluster-config.yaml): kubernetes runtime · Local Kubernetes host for the workshop stack · `kind-cluster-config.yaml`
- [Namespaces](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/k8s/namespaces/namespaces.yaml): cluster isolation · Namespace boundaries for deployable subsystems · `k8s/namespaces/namespaces.yaml`
- [MinIO](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/k8s/minio/minio-deployment.yaml): object storage · Shared data lake and artifact store · `k8s/minio/minio-deployment.yaml`
- [Postgres](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/k8s/postgres/postgres-deployment.yaml): metadata db · Backing store for Airflow and MLflow metadata · `k8s/postgres/postgres-deployment.yaml`
- [MLflow](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/k8s/mlflow/mlflow-deployment.yaml): tracking/registry · Experiment tracking and model registry service · `k8s/mlflow/mlflow-deployment.yaml`
- [Airflow](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/k8s/airflow/airflow-webserver.yaml): orchestrator · Pipeline control plane and scheduled DAG runner · `k8s/airflow/airflow-webserver.yaml`
- [Redis](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/k8s/redis/redis-deployment.yaml): online cache · Low-latency lookup store for inference time · `k8s/redis/redis-deployment.yaml`

### Pipeline

Orchestration and ML lifecycle jobs

- [Ingestion job](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/src/data/download_data.py): data pull · Downloads the IBM Telco churn dataset · `src/data/download_data.py`
- [Feature engineering](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/src/features/feature_engineering.py): transform job · Builds model-ready features from raw customer data · `src/features/feature_engineering.py`
- [Training job](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/src/training/train_model.py): model training · Trains churn models and logs runs · `src/training/train_model.py`
- [Production DAGs](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/dags/05_full_pipeline.py): airflow dags · Staged pipeline definitions for ingestion, features, training, and evaluation · `dags/05_full_pipeline.py`
- [Git DAG sync](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/k8s/airflow/airflow-git-sync-configmap.yaml): dag distribution · Git-backed DAG delivery path for Airflow · `k8s/airflow/airflow-git-sync-configmap.yaml`
- **IBM Telco data**: external source · Upstream churn dataset used by ingestion

### Applications

Training, serving, and UI code

- [Serving API](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/src/serving/app/main.py): fastapi inference · Model inference service with routing and prediction endpoints · `src/serving/app/main.py`
- [A/B router](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/src/serving/app/ab_router.py): traffic split · Champion/challenger routing logic for inference · `src/serving/app/ab_router.py`
- [Predictor](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/src/serving/app/predictor.py): model runtime · Model loading and prediction execution · `src/serving/app/predictor.py`
- [Dashboard](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/src/frontend/src/App.jsx): react ui · Client UI for predictions, model comparison, and A/B views · `src/frontend/src/App.jsx`

### Ops

Cluster lifecycle and deployment automation

- [Deploy scripts](https://github.com/rajinikanthvadla-ai/mlops-lab-workshop/blob/HEAD/scripts/deploy-all.ps1): powershell ops · Cluster creation, deployment, port-forwarding, and cleanup automation · `scripts/deploy-all.ps1`

## Connections

- Deploy scripts → Kind cluster: create/deploy · Bootstraps and manages the local cluster
- Kind cluster → Namespaces: hosts · Cluster boundary for isolated workloads
- Kind cluster → Airflow: runs · Airflow deployed into Kubernetes
- Kind cluster → MLflow: runs · MLflow deployed into Kubernetes
- Kind cluster → MinIO: runs · Shared storage service in-cluster
- Kind cluster → Postgres: runs · Metadata database in-cluster
- Kind cluster → Redis: runs · Inference cache in-cluster
- Kind cluster → Serving API: runs · Model serving workload in-cluster
- Kind cluster → Dashboard: runs · Dashboard workload in-cluster
- Git DAG sync → Airflow: syncs · Alternative DAG delivery mechanism
- Airflow → Production DAGs: orchestrates · Schedules and coordinates pipeline stages
- Production DAGs → Ingestion job: executes · Ingestion stage definition
- Production DAGs → Feature engineering: executes · Feature engineering stage definition
- Production DAGs → Training job: executes · Training stage definition
- Ingestion job → IBM Telco data: pulls · Fetches the external churn dataset
- Ingestion job → MinIO: writes raw · Stores raw data in the lake
- Feature engineering → MinIO: reads/writes · Consumes raw data and persists processed features
- Training job → MinIO: reads features · Loads processed training data
- Training job → MLflow: logs · Stores metrics, artifacts, and model versions
- MLflow → Postgres: metadata · Tracks runs and registry state
- MLflow → MinIO: artifacts · Persists model artifacts and files
- Serving API → MLflow: loads model · Resolves approved model versions
- Serving API → Redis: queries · Uses online cache for inference lookups
- A/B router → Serving API: inside · Routing logic embedded in the serving service
- Predictor → Serving API: inside · Prediction engine embedded in the serving service
- Dashboard → Serving API: calls api · Consumes prediction and experiment endpoints
