How rajdeep712/mlops-capstone-project works
Purpose and Entry Points
This repository implements an end-to-end Python MLOps workflow: reproducible model training, experiment/model tracking, containerized Flask inference, and AWS/Kubernetes delivery. Primary local entry points are DVC reproduction via dvc.yaml, the prediction service in flask_app/app.py, and model promotion in scripts/promote_model.py.
Reproducible Training Orchestration
DVC defines the training pipeline and dependency graph in dvc.yaml, with run parameters centralized in params.yaml and locked outputs/dependencies in dvc.lock. DVC configuration in .dvc/config establishes versioned data/pipeline state and is intended to use an S3 remote.
Data Acquisition and Preparation
The data boundary begins with ingestion and external source access: src/data/data_ingestion.py, src/connections/s3_connection.py, and src/connections/ssms_connection.py. Raw data then passes through preprocessing and transformation stages in src/data/data_preprocessing.py and src/features/feature_engineering.py, producing model-ready features.
Model Lifecycle
Training, evaluation, and registration are separated into dedicated pipeline stages: src/model/model_building.py, src/model/model_evaluation.py, and src/model/register_model.py. The intended control flow is features → trained scikit-learn model → metrics/artifacts logged to MLflow/Dagshub → selected model registered or promoted. scripts/promote_model.py is the explicit operational promotion boundary.
Online Inference Service
flask_app/app.py is the serving runtime and HTTP entry point. It loads the deployed model and delegates request/input handling or transformations to flask_app/processing_utility.py; flask_app/templates/index.html supplies the browser-facing interface. Training code and online serving are separate boundaries: serving consumes an already-selected model rather than executing DVC stages.
Packaging, Deployment, and CI/CD
Dockerfile packages the Flask service for deployment. deployment.yaml describes Kubernetes deployment/service infrastructure, intended for Amazon EKS and external exposure. GitHub Actions at .github/workflows/ci.yaml automates validation and the delivery path described by the project: tests/DVC work, image build, ECR publication, and EKS deployment. AWS integrations span S3 for DVC data, ECR for images, and EKS for runtime orchestration.
Observability and Validation
The deployed Flask API is intended to expose metrics for Prometheus, with Grafana consuming Prometheus data for dashboards; their infrastructure definitions are not present in the tree. Tests focus on the main execution boundaries: inference behavior in tests/test_flask_app.py and model behavior in tests/test_model.py. Python 3.10 is the primary runtime; dependency manifests are requirements.txt and flask_app/requirements.txt.
Main entry points: app.py Flask API · dvc.yaml DVC pipeline · ci.yaml GitHub Actions · model_building.py Model training
Components 22
Training Pipeline
Reproducible DVC-managed model lifecycle.
- DVC pipeline · Defines reproducible training stages.
dvc.yaml - Run parameters
params.yaml - DVC state · Locks dependencies and configures versioned data state.
dvc.lock - S3 data remote · DVC remote and external data source.
- Data ingestion
src/data/data_ingestion.py - AWS connections · S3 and SSM access adapters.
src/connections/s3_connection.py - Preprocessing
src/data/data_preprocessing.py - Feature engineering · Produces model-ready features.
src/features/feature_engineering.py - Model training · Builds a scikit-learn model.
src/model/model_building.py - Model evaluation
src/model/model_evaluation.py - MLflow / Dagshub · Receives metrics and model artifacts.
- Model registration
src/model/register_model.py - Model promotion · Selects the model for deployment.
scripts/promote_model.py
Inference Service
Separate online Flask runtime.
- Flask API · Loads the selected model and serves predictions.
flask_app/app.py - Request processing · Handles inference inputs and transformations.
flask_app/processing_utility.py - Browser interface
flask_app/templates/index.html
Delivery
Container build and AWS Kubernetes release path.
- Service image
Dockerfile - GitHub Actions · Validates, builds, publishes, and deploys.
.github/workflows/ci.yaml - Amazon ECR
- Amazon EKS · Runs the externally exposed Flask service.
deployment.yaml
Other components
- Prometheus · Scrapes deployed API metrics.
- Grafana
Connections 25
- Run parameters → DVC pipeline: configures stages
- DVC state → DVC pipeline: locks runs
- DVC pipeline → Data ingestion: runs
- S3 data remote → Data ingestion: source data
- AWS connections → S3 data remote: accesses
- Data ingestion → Preprocessing: raw data
- Preprocessing → Feature engineering: prepared data
- Feature engineering → Model training: features
- Model training → Model evaluation: trained model
- Model training → MLflow / Dagshub: artifacts
- Model evaluation → MLflow / Dagshub: metrics
- Model evaluation → Model registration: validated model
- Model registration → Model promotion: registered model
- Model promotion → Flask API: selected model
- Browser interface → Flask API: requests
- Flask API → Request processing: delegates inputs
- Service image → Flask API: packages
- GitHub Actions → DVC pipeline: validates pipeline
- GitHub Actions → Service image: builds image
- Service image → Amazon ECR: publishes image
- GitHub Actions → Amazon EKS: deploys
- Amazon ECR → Amazon EKS: image pull
- Amazon EKS → Flask API: hosts
- Prometheus → Flask API: scrapes metrics
- Grafana → Prometheus: queries metrics