# rajinikanthvadla-ai/project-2-e2e-aws-mlops architecture

> Architecture diagram of the GitHub repository rajinikanthvadla-ai/project-2-e2e-aws-mlops, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops
- Repository: https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops

## Overview

##### Project Type and Purpose
- This is a full-stack MLOps project demonstrating end-to-end CI/CD for a machine learning workflow on AWS.
- Key purposes:
- Automate data ingestion and manifest creation (Lambda + S3)
- CI stage (lint, test, build) via AWS CodePipeline/CodeBuild
- SageMaker Pipeline for data preprocessing, training, evaluation
- Model deployment to a SageMaker real-time endpoint (via CDK)
- Online model consumption via a React web app behind API Gateway

##### File Structure
- Top-level directories:
- infrastructure – CDK app defining CodePipeline, Lambda & S3 constructs
- ml_pipeline – Python code defining the SageMaker Pipeline stages
- model_deploy – CDK app for deploying prediction endpoint & Lambda layers
- consumers/online – React frontend + CDK infra to host web app + API
- src & tests – core Python ML code (preprocess, transform, evaluate) and pytest
- buildspecs & scripts – CI/CD build configurations and bash helpers
- configuration – projectConfig.json drives repoType, IP whitelist

##### README Insights
- Confirms AWS services used: S3 (raw data + manifest), Lambda (data monitor), CodePipeline (CI/CD), SageMaker (training + endpoint), API Gateway, CloudFront+S3 (frontend).

##### Main Components & Interactions
a. Source Control
- GitHub or AWS CodeCommit repo (driven by configuration/projectConfig.json)
b. CI/CD Orchestration
- CodePipeline with stages:
– Source
– CI (CodeBuild running buildspecs/build.yml: tests, lint, packaging)
– SageMaker Pipeline execution
– Deploy (manual approval → CDK deploy of model_deploy stack)
c. Data Ingestion
- S3 Data Bucket (raw CSV uploads)
- Lambda function (infrastructure/functions/dataSourceMonitor) writes Data Manifest to S3
d. SageMaker Pipeline
- Defined in ml_pipeline/pipeline.py + get_pipeline_definition.py
- Steps: preprocess (sklearn), train (XGBoost), evaluate, quality gate
e. Model Deployment
- CDK stack in model_deploy/lib/modelDeploymentStack.ts
- Creates SageMaker real-time endpoint using the pipeline model
f. Model Consumption (Online)
- API Gateway + Lambda or direct integration to SageMaker endpoint
- Static React single-page app (in consumers/online/packages/website) hosted via CloudFront + S3
g. Configuration & Scripts
- scripts/bootstrap.sh, test.sh, deploy.sh orchestrate bootstrap, test, deploy
- buildspecs/*.yml define CodeBuild stages

##### Architectural Patterns & Principles
- Infrastructure-as-Code: AWS CDK (TypeScript) defines all AWS resources
- Modular monorepo: separate packages for API, data types, infrastructure, website
- Event-driven: S3 uploads → Lambda → manifest → CodePipeline trigger
- Pipeline-as-Code: SageMaker Pipeline defined in Python
- Separation of concerns:
- Infra CDK apps (infrastructure, model_deploy, consumers/online)
- ML code (src, ml_pipeline)
- Frontend (React)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Source Control
    subgraph "Source Control"
        SC["GitHub / CodeCommit"]:::infra
    end

    %% CI/CD Layer
    subgraph "CI/CD Orchestration" 
        CP["CodePipeline"]:::infra
        CB["CodeBuild"]:::compute
        BS1["buildspecs/build.yml"]:::infra
        BS2["buildspecs/pipeline.yml"]:::infra
        BS3["buildspecs/deploy.yml"]:::infra
        GH[".github/workflows/ci.yml"]:::infra
    end

    %% Data Layer
    subgraph "Data Ingestion" 
        S3Raw["S3 Raw Data Bucket"]:::data
        Lambda["DataSourceMonitor Lambda"]:::compute
        S3Manifest["S3 Manifest Bucket"]:::data
    end

    %% ML Pipeline Layer
    subgraph "SageMaker Pipeline Definition"
        PipelineDef["ml_pipeline/get_pipeline_definition.py"]:::ml
        Pipeline["ml_pipeline/pipeline.py"]:::ml
        RunPipe["ml_pipeline/run_pipeline.py"]:::ml
        Utils["ml_pipeline/_utils.py"]:::ml
    end

    %% Core ML Code
    subgraph "Core ML Code"
        Preproc["src/preprocess.py"]:::ml
        Transform["src/transform.py"]:::ml
        Evaluate["src/evaluate.py"]:::ml
        TestPre["tests/test_preprocess.py"]:::ml
    end

    %% Model Deployment Layer
    subgraph "Model Deployment (CDK)" 
        MDStack["model_deploy/lib/modelDeploymentStack.ts"]:::infra
        CR["model_deploy/customResources/pipelineModel/index.ts"]:::infra
        Layer["model_deploy/layers/PipelineModelFunctionLayer/nodejs/package.json"]:::infra
        Endpoint["SageMaker Real-Time Endpoint"]:::ml
    end

    %% Online Consumption Layer
    subgraph "Online Model Consumption"
        subgraph "Website & API Infra"
            WAIC["consumers/online/packages/infrastructure/lib/websiteAPIConstruct.ts"]:::infra
            WC["consumers/online/packages/infrastructure/lib/websiteConstruct.ts"]:::infra
            WB["consumers/online/packages/infrastructure/bin/app.ts"]:::infra
        end
        subgraph "Data Clients"
            DAC["consumers/online/packages/data-api/src/index.ts"]:::frontend
            DTC["consumers/online/packages/data-type/src/index.ts"]:::frontend
        end
        subgraph "React SPA"
            Web["consumers/online/packages/website/src/index.tsx"]:::frontend
            Components["consumers/online/packages/website/src/components"]:::frontend
            Containers["consumers/online/packages/website/src/containers"]:::frontend
            HTML["consumers/online/packages/website/public/index.html"]:::frontend
        end
        APIGW["API Gateway"]:::compute
        APILambda["API Lambda"]:::compute
    end

    %% Configuration & Scripts
    subgraph "Configuration & Scripts"
        Config["configuration/projectConfig.json"]:::infra
        Header["configuration/header.js"]:::infra
        BScripts["scripts/bootstrap.sh"]:::infra
        TScripts["scripts/test.sh"]:::infra
        DScripts["scripts/deploy.sh"]:::infra
        RP["scripts/run_pipeline.sh"]:::infra
        CDM["scripts/copy_data_manifest.sh"]:::infra
    end

    %% Connections
    SC -->|"Push Code"| CP
    Config --> CP
    CP -->|"Source to Build"| CB
    CB -->|uses| BS1 & BS2 & BS3
    CB -->|"CI Tests"| Preproc & Transform & Evaluate & TestPre
    S3Raw -->|"Upload CSV"| Lambda
    Lambda -->|"Write Manifest"| S3Manifest
    S3Manifest -->|"Trigger"| CP
    CP -->|"Run SageMaker"| Pipeline
    PipelineDef --> Pipeline
    RunPipe --> Pipeline
    Utils --> Pipeline
    Pipeline -->|"Output Model"| MDStack
    MDStack -->|"Deploy Endpoint"| Endpoint
    Endpoint -->|"Invoke"| APIGW
    WAIC --> APIGW
    APIGW --> APILambda
    APILambda --> Endpoint
    APIGW -->|"Serve"| Web
    DAC --> Web
    DTC --> Web

    %% Click Events
    click CP "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/infrastructure/lib/codePipelineConstruct.ts"
    click BS1 "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/buildspecs/build.yml"
    click BS2 "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/buildspecs/pipeline.yml"
    click BS3 "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/buildspecs/deploy.yml"
    click GH "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/.github/workflows/ci.yml"
    click Lambda "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/infrastructure/functions/dataSourceMonitor/src/index.ts"
    click S3Raw "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/infrastructure/lib/dataSourceConstruct.ts"
    click Pipeline "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/ml_pipeline/pipeline.py"
    click PipelineDef "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/ml_pipeline/get_pipeline_definition.py"
    click RunPipe "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/ml_pipeline/run_pipeline.py"
    click Utils "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/ml_pipeline/_utils.py"
    click Preproc "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/src/preprocess.py"
    click Transform "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/src/transform.py"
    click Evaluate "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/src/evaluate.py"
    click TestPre "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/tests/test_preprocess.py"
    click MDStack "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/model_deploy/lib/modelDeploymentStack.ts"
    click CR "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/model_deploy/customResources/pipelineModel/index.ts"
    click Layer "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/model_deploy/layers/PipelineModelFunctionLayer/nodejs/package.json"
    click WAIC "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/consumers/online/packages/infrastructure/lib/websiteAPIConstruct.ts"
    click WC "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/consumers/online/packages/infrastructure/lib/websiteConstruct.ts"
    click WB "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/consumers/online/packages/infrastructure/bin/app.ts"
    click DAC "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/consumers/online/packages/data-api/src/index.ts"
    click DTC "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/consumers/online/packages/data-type/src/index.ts"
    click Web "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/consumers/online/packages/website/src/index.tsx"
    click Components "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/tree/main/consumers/online/packages/website/src/components"
    click Containers "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/tree/main/consumers/online/packages/website/src/containers"
    click HTML "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/consumers/online/packages/website/public/index.html"
    click Config "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/configuration/projectConfig.json"
    click Header "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/configuration/header.js"
    click BScripts "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/scripts/bootstrap.sh"
    click TScripts "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/scripts/test.sh"
    click DScripts "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/scripts/deploy.sh"
    click RP "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/scripts/run_pipeline.sh"
    click CDM "https://github.com/rajinikanthvadla-ai/project-2-e2e-aws-mlops/blob/main/scripts/copy_data_manifest.sh"

    %% Styles
    classDef data fill:#D0E8FF,stroke:#0366d6,color:#0366d6
    classDef compute fill:#FFE4B5,stroke:#FF8C00,color:#8B4513
    classDef ml fill:#E2F0D9,stroke:#228B22,color:#006400
    classDef frontend fill:#E9D5FF,stroke:#800080,color:#4B0082
    classDef infra fill:#F2F2F2,stroke:#A9A9A9,color:#696969
```
