How rajinikanthvadla-ai/project-2-e2e-aws-mlops works
Project Type and Purpose
• This is a full-stack MLOps project demonstrating end-to-end CI/CD for a machine learning workflow on AWS.
• Key purposes:
• Automate data ingestion and manifest creation (Lambda + S3)
• CI stage (lint, test, build) via AWS CodePipeline/CodeBuild
• SageMaker Pipeline for data preprocessing, training, evaluation
• Model deployment to a SageMaker real-time endpoint (via CDK)
• Online model consumption via a React web app behind API Gateway
File Structure
• Top-level directories:
• infrastructure – CDK app defining CodePipeline, Lambda & S3 constructs
• ml_pipeline – Python code defining the SageMaker Pipeline stages
• model_deploy – CDK app for deploying prediction endpoint & Lambda layers
• consumers/online – React frontend + CDK infra to host web app + API
• src & tests – core Python ML code (preprocess, transform, evaluate) and pytest
• buildspecs & scripts – CI/CD build configurations and bash helpers
• configuration – projectConfig.json drives repoType, IP whitelist
README Insights
• Confirms AWS services used: S3 (raw data + manifest), Lambda (data monitor), CodePipeline (CI/CD), SageMaker (training + endpoint), API Gateway, CloudFront+S3 (frontend).
Main Components & Interactions
a. Source Control
• GitHub or AWS CodeCommit repo (driven by configuration/projectConfig.json)
b. CI/CD Orchestration
• CodePipeline with stages:
– Source
– CI (CodeBuild running buildspecs/build.yml: tests, lint, packaging)
– SageMaker Pipeline execution
– Deploy (manual approval → CDK deploy of model_deploy stack)
c. Data Ingestion
• S3 Data Bucket (raw CSV uploads)
• Lambda function (infrastructure/functions/dataSourceMonitor) writes Data Manifest to S3
d. SageMaker Pipeline
• Defined in ml_pipeline/pipeline.py + get_pipeline_definition.py
• Steps: preprocess (sklearn), train (XGBoost), evaluate, quality gate
e. Model Deployment
• CDK stack in model_deploy/lib/modelDeploymentStack.ts
• Creates SageMaker real-time endpoint using the pipeline model
f. Model Consumption (Online)
• API Gateway + Lambda or direct integration to SageMaker endpoint
• Static React single-page app (in consumers/online/packages/website) hosted via CloudFront + S3
g. Configuration & Scripts
• scripts/bootstrap.sh, test.sh, deploy.sh orchestrate bootstrap, test, deploy
• buildspecs/*.yml define CodeBuild stages
Architectural Patterns & Principles
• Infrastructure-as-Code: AWS CDK (TypeScript) defines all AWS resources
• Modular monorepo: separate packages for API, data types, infrastructure, website
• Event-driven: S3 uploads → Lambda → manifest → CodePipeline trigger
• Pipeline-as-Code: SageMaker Pipeline defined in Python
• Separation of concerns:
• Infra CDK apps (infrastructure, model_deploy, consumers/online)
• ML code (src, ml_pipeline)
• Frontend (React)