# baldcodr/e2e-mlops-infra-aws architecture

> Architecture diagram of the GitHub repository baldcodr/e2e-mlops-infra-aws, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/baldcodr/e2e-mlops-infra-aws
- Repository: https://github.com/baldcodr/e2e-mlops-infra-aws

## Overview

##### Main Components
- Source Code & Configuration: At the root, you have configuration files (e.g., buildspec.yml, setup.py, tox.ini), version control files (e.g., .gitignore) and high-level CI configuration within the “ci” folder. These components show the automated build, test, and deploy aspects.
- CI/CD Pipeline: The “ci” and “deploy” directories contain scripts and templates (like catalog.yml, codeconnection.yml, template.yml, and build.py) to manage integrations, builds, and deployments. Include both production and testing-related deployment paths (noted by deploy/prod-config.json, staging-config.json, and deploy/test).
- ML Pipelines: The “pipelines” directory contains modules specifically for the machine learning workflows – for instance, the “abalone” pipeline includes modules for preprocessing, evaluation, and orchestrating the pipeline. There are additional modules (e.g., get_pipeline_definition.py, run_pipeline.py) that likely interface with AWS services (SageMaker Pipelines) to define or execute these workflows.
- Experimentation/Notebooks: The “sagemaker-pipelines-project.ipynb” indicates an interactive component possibly used by data scientists to experiment and iterate over models before pushing changes.
- Tests: The “tests” folder shows that there is a testing strategy in place to verify pipeline functionality.

##### Relationships and Interactions
- The Source Code repository is the starting point, where developers check in code and configuration.
- CI/CD systems (observed under “ci” and “deploy”) automatically trigger builds and tests (using buildspec files) whenever there’s a commit. They integrate with AWS deployment pipelines to manage environments (test, staging, production).
- ML Pipelines, developed and defined in the “pipelines” modules, manage different aspects of the machine learning process (data preprocessing, evaluation, overall orchestration). These pipelines likely interact with AWS’s SageMaker for training and deployment, as suggested by the notebook and naming.
- The experimentation notebook serves as an interactive development interface and might feed updated configurations or models into the pipeline.
- Tests validate pipeline code ensuring the processes are robust before triggering production deployment.

##### Architectural Patterns & Important Technologies
- MLOps/DevOps pattern for CI/CD is central. The build and deploy scripts indicate an automated and repeatable deployment process.
- Use of pipeline modules separates concerns – for example, a clear separation between data preprocessing, evaluation, and pipeline orchestration. This aligns with modular design principles.
- AWS-specific configurations and deployment templates (e.g., endpoint-config-template.yml) indicate an architecture that leverages managed cloud services, such as Amazon SageMaker or other AWS deployment tools.
- The project is configured with testing and continuous integration utilities (e.g., tox.ini, .coveragerc), stressing the importance of quality and automation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Source Code & Configuration Subgraph
    subgraph "Source Code & Configuration"
        SC1["buildspec.yml"]:::source
        SC2["setup.py"]:::source
        SC3["tox.ini"]:::source
        SC4[".gitignore"]:::source
        SC5["setup.cfg"]:::source
        SC6[".coveragerc"]:::source
        SC7["README.md"]:::source
    end

    %% CI/CD Pipeline Subgraph
    subgraph "CI/CD Pipeline"
        subgraph "CI"
            CI1["ci/catalog.yml"]:::cicd
            CI2["ci/codeconnection.yml"]:::cicd
            CI3["ci/template.yml"]:::cicd
        end
        subgraph "Deploy"
            CD1["deploy/build.py"]:::cicd
            CD2["deploy/buildspec.yml"]:::cicd
            CD3["deploy/endpoint-config-template.yml"]:::cicd
            CD4["deploy/prod-config.json"]:::cicd
            CD5["deploy/staging-config.json"]:::cicd
            subgraph "Deploy Test"
                DT1["deploy/test/buildspec.yml"]:::cicd
                DT2["deploy/test/test.py"]:::cicd
            end
        end
    end

    %% ML Pipelines Subgraph
    subgraph "ML Pipelines"
        subgraph "Core Pipelines"
            ML1["pipelines/__init__.py"]:::ml
            ML2["pipelines/__version__.py"]:::ml
            ML3["pipelines/_utils.py"]:::ml
            ML4["pipelines/get_pipeline_definition.py"]:::ml
            ML5["pipelines/run_pipeline.py"]:::ml
        end
        subgraph "Abalone Pipeline"
            AB1["pipelines/abalone/__init__.py"]:::ml
            AB2["pipelines/abalone/evaluate.py"]:::ml
            AB3["pipelines/abalone/pipeline.py"]:::ml
            AB4["pipelines/abalone/preprocess.py"]:::ml
        end
    end

    %% Experimentation/Notebooks Subgraph
    subgraph "Experimentation"
        NB["sagemaker-pipelines-project.ipynb"]:::notebook
    end

    %% Test Suite Subgraph
    subgraph "Test Suite"
        TS["tests/test_pipelines.py"]:::test
    end

    %% AWS Deployment (SageMaker) Node
    AWS["AWS Deployment (SageMaker)"]:::aws

    %% Relationships
    SC1 -->|"commit_trigger"| CI1
    SC1 -->|"update_pipeline"| ML1
    NB -->|"experiment_feed"| ML1
    TS -->|"validate"| ML1
    TS -->|"feedback"| CI1
    CI1 -->|"deploy_code"| AWS
    ML1 -->|"trigger_training"| AWS

    %% Click Events for Source Code & Configuration
    click SC1 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/buildspec.yml"
    click SC2 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/setup.py"
    click SC3 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/tox.ini"
    click SC4 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/.gitignore"
    click SC5 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/setup.cfg"
    click SC6 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/.coveragerc"
    click SC7 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/README.md"

    %% Click Events for CI/CD Pipeline (CI)
    click CI1 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/ci/catalog.yml"
    click CI2 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/ci/codeconnection.yml"
    click CI3 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/ci/template.yml"
    
    %% Click Events for CI/CD Pipeline (Deploy)
    click CD1 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/deploy/build.py"
    click CD2 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/deploy/buildspec.yml"
    click CD3 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/deploy/endpoint-config-template.yml"
    click CD4 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/deploy/prod-config.json"
    click CD5 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/deploy/staging-config.json"
    click DT1 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/deploy/test/buildspec.yml"
    click DT2 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/deploy/test/test.py"

    %% Click Events for ML Pipelines (Core)
    click ML1 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/pipelines/__init__.py"
    click ML2 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/pipelines/__version__.py"
    click ML3 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/pipelines/_utils.py"
    click ML4 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/pipelines/get_pipeline_definition.py"
    click ML5 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/pipelines/run_pipeline.py"

    %% Click Events for ML Pipelines (Abalone)
    click AB1 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/pipelines/abalone/__init__.py"
    click AB2 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/pipelines/abalone/evaluate.py"
    click AB3 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/pipelines/abalone/pipeline.py"
    click AB4 "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/pipelines/abalone/preprocess.py"

    %% Click Event for Experimentation/Notebooks
    click NB "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/sagemaker-pipelines-project.ipynb"

    %% Click Event for Test Suite
    click TS "https://github.com/baldcodr/e2e-mlops-infra-aws/blob/main/tests/test_pipelines.py"

    %% Styles
    classDef source fill:#dfefff,stroke:#333,stroke-width:2px;
    classDef cicd fill:#ffefc6,stroke:#333,stroke-width:2px;
    classDef ml fill:#d1f1c3,stroke:#333,stroke-width:2px;
    classDef notebook fill:#ffe0e0,stroke:#333,stroke-width:2px;
    classDef test fill:#e0e0ff,stroke:#333,stroke-width:2px;
    classDef aws fill:#fff2cc,stroke:#333,stroke-width:2px,stroke-dasharray: 5,5;
```
