How aaditya025/mlops_heart_disease_project works
1. This repository is organized around two clear planes: an offline ML lifecycle and an online inference/ops stack. The offline side prepares data, trains models, and stores artifacts; the online side serves predictions, exposes metrics, and supports deployment/monitoring.
2. The dataset lives in data/ and is based on the UCI Heart Disease set. The repo includes raw and processed CSV-like files, plus a cleaned heart_disease.csv, so the training flow is file-based rather than database-backed.
3. The offline pipeline is implemented in src/download_data.py, src/eda.py, and src/train.py. The flow is: fetch/prepare data → perform EDA → train candidate classifiers → evaluate metrics → persist the chosen model artifacts.
4. Training is tracked with MLflow, which is materialized in the mlruns/ directory. Each run stores parameters, metrics, and serialized model artifacts, making experiment comparison and reproducibility part of the architecture rather than an external concern.
5. The production model artifacts are also checked into models/ as best_model.pkl, pipeline.pkl, scaler.pkl, and feature_names.pkl. This indicates the serving layer depends on prebuilt serialized artifacts, not live model training.
6. The serving boundary is src/app.py, a FastAPI application. It exposes /predict, /health, /metrics, /model/info, and documentation endpoints, and likely loads the trained pipeline at startup to transform input clinical features into a prediction response.
7. The repository also includes a small static frontend under src/static/ (index.html, script.js, style.css), which acts as a lightweight UI for entering patient features and calling the prediction API. This UI sits beside, not inside, the model logic.
8. Observability is built into the service through Prometheus metrics and Grafana dashboards. config/prometheus.yml defines scraping, while config/grafana/ contains datasource and dashboard provisioning plus a model_metrics.json dashboard definition. This means monitoring is intended to be fully provisioned from config.
9. A traffic generator (generate_traffic.py) and traffic.log suggest the system can simulate prediction load for testing the metrics pipeline and dashboard behavior. This matters because monitoring is part of the deliverable, not just the application.
10. Containerization is a first-class boundary. The Dockerfile packages the service, docker-compose.yaml assembles the local stack, and start.sh is the convenience entry point mentioned in the README for bringing up API + Prometheus + Grafana + traffic generation together.
11. Kubernetes deployment is defined separately in k8s/deployment.yaml, k8s/ingress.yaml, and k8s/monitoring.yaml. That split implies three deployment concerns: app pods, external access routing, and observability resources.
12. CI/CD is handled by GitHub Actions in .github/workflows/ci-cd.yml. The pipeline gates the repo with linting, tests, training, Docker build checks, security scanning, and deployment, so the software lifecycle is automated end-to-end.
13. Tests are split by concern in tests/: test_data.py, test_model.py, and test_api.py. That mirrors the architecture: data validation, model behavior, and service contract are verified independently.
14. There is no evidence of a backing database, queue, or external model server. The system’s core state is file-based: datasets, MLflow runs, and pickled artifacts on disk, which keeps the architecture simple but ties inference to packaged model files.
15. For diagramming, the key data flow is: dataset files → EDA/training → MLflow + serialized model artifacts → FastAPI inference service → Prometheus/Grafana monitoring. Deployment then wraps the same service in Docker and Kubernetes.
Main entry points: app.py FastAPI app · train.py Train model · ci-cd.yml CI/CD · prometheus.yml Prometheus cfg
Components 20
Offline ML
Data prep, EDA, training, and experiment tracking
- Heart data · UCI Heart Disease CSV-style inputs and cleaned dataset
data/heart_disease.csv - Fetch data · Downloads or prepares the source heart disease data
src/download_data.py - EDA · Explores and summarizes the dataset before training
src/eda.py - Train model · Builds candidate classifiers and evaluates metrics
src/train.py - MLflow runs · Tracks parameters, metrics, and serialized model artifacts
mlruns/551504309071015152/meta.yaml - Model files · Checked-in pipeline, scaler, feature names, and best model
models/pipeline.pkl
Online Serving
API, UI, and prediction runtime
- FastAPI app · Prediction API with health, metrics, and model info endpoints
src/app.py - Static UI · Simple browser interface for entering patient features and calling the API
src/static/index.html - Inference path · Loads the packaged pipeline and transforms inputs into predictions
Ops & Deploy
Monitoring, traffic generation, containers, and cluster deployment
- Prometheus cfg · Scrape configuration for service metrics collection
config/prometheus.yml - Grafana cfg · Datasource and dashboard provisioning for monitoring
config/grafana/provisioning/dashboards/dashboard.yml - Metrics dashboard · Model and service metrics visualization
config/grafana/dashboards/model_metrics.json - Load generator · Simulates prediction traffic to exercise monitoring
generate_traffic.py - Compose stack · Brings up API, monitoring, and optional traffic together
docker-compose.yaml - Service image · Packages the API for container deployment
Dockerfile - App deploy · Kubernetes deployment for the service pods
k8s/deployment.yaml - Ingress · External routing into the service
k8s/ingress.yaml - K8s monitoring · Cluster-side observability resources
k8s/monitoring.yaml
Connections 25
- Heart data → Fetch data: source data · Raw and cleaned files feed preparation
- Fetch data → EDA: prepared data · Prepared dataset is inspected in EDA
- EDA → Train model: insights · Exploration informs training and feature handling
- Heart data → Train model: training set · Training consumes the file-based heart disease dataset
- Train model → MLflow runs: log runs · Metrics, params, and artifacts are recorded in MLflow
- Train model → Model files: persist model · Chosen model artifacts are stored for serving
- MLflow runs → Model files: best artifacts · Registered model outputs mirror the packaged artifacts
- Model files → FastAPI app: load model · FastAPI loads the serialized pipeline and auxiliaries
- FastAPI app → Inference path: serve predictions · Request handling enters the inference path
- Static UI → FastAPI app: call predict · Frontend submits feature payloads to the API
- FastAPI app → Prometheus cfg: emit metrics · Service exposes metrics for scraping
- Prometheus cfg → Metrics dashboard: scrape data · Prometheus feeds monitoring dashboards
- Grafana cfg → Metrics dashboard: provision · Dashboards and datasource are provisioned from config
- Load generator → FastAPI app: load test · Synthetic traffic drives prediction and metrics activity
- Compose stack → FastAPI app: orchestrate · Compose starts the service container
- Compose stack → Prometheus cfg: orchestrate · Compose also brings up monitoring components
- Compose stack → Grafana cfg: orchestrate · Grafana is part of the local stack
- Service image → FastAPI app: package · The API is containerized from the project image
- App deploy → FastAPI app: deploy · Kubernetes deployment runs the service
- Ingress → FastAPI app: route · Ingress exposes the service externally
- K8s monitoring → Prometheus cfg: monitor · Cluster monitoring aligns with Prometheus scraping
- CI/CD → Tests: run tests · Pipeline executes automated validation
- CI/CD → Train model: train check · Pipeline includes training-related verification
- CI/CD → Service image: build image · Pipeline validates container build
- CI/CD → App deploy: deploy check · Pipeline covers deployment artifacts