# hdegen/mlops-end-to-end architecture

> Architecture diagram of the GitHub repository hdegen/mlops-end-to-end, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hdegen/mlops-end-to-end
- Repository: https://github.com/hdegen/mlops-end-to-end

## Overview

##### Project type and purpose
- This is a full-stack MLOps template for building, tracking, and deploying machine-learning pipelines on AWS.
- It standardizes project structure, orchestrates data→feature→model steps via MLflow, DVC/MLflow data versioning, containerizes with Docker, and schedules via Apache Airflow on cloud compute.

##### File structure
- Top-level:
- dag.py – example Airflow DAG
- Dockerfile, build/run scripts – containerization
- data/, notebooks/ – raw data placeholders and exploratory notebooks
- src/ – source code subdivided into data, features, models, lib, examples, plus MLproject, conda.yaml, main.py
- src/data: data ingestion and cleaning (SQL queries, S3 reads)
- src/features: feature engineering logic
- src/models: model training and prediction entry points
- src/lib: utility functions and AWS/MLflow helper code
- src/examples: CLI usage examples
- src/main.py: orchestrates MLflow steps using Click for parameter parsing

##### Extract key architectural insights from the README
- Uses MLflow to define a reproducible pipeline (MLproject file) with steps: make_data → build_features → train_model → predict
- Tracks parameters, metrics, and artifacts (including versioned data) via MLflow backed by an AWS-hosted database and S3 artifact store
- Containerization: builds a conda-based Docker image, pushes to AWS ECR via CI/CD (e.g., Jenkins)
- Scheduling: defines an Airflow DAG to run each pipeline step on cloud resources (EC2/EKS, CPU/GPU)
- Optional DVC for data versioning—though MLflow is used here to store data snapshots in S3

1. Local Development Workstation (IDE, Conda env)
2. CI/CD Pipeline (Jenkins/GitHub Actions)
3. Docker Registry (AWS ECR)
4. Orchestration Layer (Apache Airflow on EKS or EC2)
5. Compute Tasks (container-based jobs for data, features, training, predict)
6. MLflow Tracking Server (backed by RDS/PostgreSQL)
7. Artifact Store (AWS S3 buckets for raw data, processed data, models)
8. Metadata Store (MLflow metadata database)
9. External Services (AWS S3, Athena, other AWS APIs)

- Developer → commit code → CI/CD → build Docker image → push to ECR
- Airflow scheduler → pull latest image from ECR → spin up tasks for each pipeline step
- Pipeline tasks → read raw data from S3 (or query Athena) → write cleaned data/features back to S3
- Training task → log params/metrics/models to MLflow Tracking Server → store artifacts in S3
- Predict task → retrieve model from MLflow registry → write predictions to S3 or downstream system

c. Architectural patterns and principles:
- Modular pipeline steps (decoupled via MLproject entry points)
- Containerization for reproducibility and portability
- Orchestration via DAG for flexible scheduling and scaling
- Separation of concerns: code (src/) vs. data (S3) vs. compute (containers) vs. metadata (RDS)
- Infrastructure as code/scripts: Dockerfile, build/run scripts, conda.yaml

d. Technologies, frameworks, libraries:
- Python, Click for CLI, MLflow for experiment tracking and model registry
- Docker + AWS ECR for container images
- Apache Airflow on AWS (EC2/EKS) for scheduling
- AWS S3 for data and model artifact storage
- AWS RDS/PostgreSQL for MLflow backend store
- (Optional) DVC for data versioning

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Local Development
    subgraph "Local Development Workstation"
        direction TB
        Conda["conda.yaml"]:::code
        Req["requirements.txt"]:::code
    end

    %% CI/CD
    subgraph "CI/CD Pipeline" 
        direction TB
        Build["build_docker.sh"]:::ci
        Run["run_docker.sh"]:::ci
        RunCloud["run_docker_cloud.sh"]:::ci
    end

    %% Containerization & Registry
    subgraph "Containerization & Registry"
        direction TB
        Dockerf["Dockerfile"]:::code
        ECR["AWS ECR"]:::storage
    end

    %% Orchestration
    subgraph "Orchestration Layer"
        direction TB
        Airflow["Apache Airflow (dag.py)"]:::compute
    end

    %% Pipeline Tasks
    subgraph "Pipeline Tasks"
        direction TB
        Data["Data Ingestion (make_data.py)"]:::compute
        SQL["SQL Query (query.sql)"]:::code
        Features["Feature Eng. (build_features.py)"]:::compute
        Train["Model Training (train_model.py)"]:::compute
        Predict["Prediction (predict.py)"]:::compute
    end

    %% MLflow Pipeline Definition
    subgraph "MLflow Pipeline Definition"
        direction TB
        MLproject["MLproject"]:::ci
        Main["main.py"]:::ci
    end

    %% Utilities & Examples & Tests
    subgraph "Utilities & Examples & Tests"
        direction TB
        Utils["utils.py"]:::code
        Example["example.sh"]:::code
        Setup["setup.py"]:::code
        RunTest["run_test.sh"]:::code
        TestPy["test.py"]:::code
    end

    %% Storage & Metadata
    subgraph "Storage & Metadata"
        direction TB
        S3["AWS S3"]:::storage
        MetadataDB["RDS/PostgreSQL"]:::storage
    end

    %% External Services
    subgraph "External Services"
        direction TB
        Athena["Athena & AWS APIs"]:::external
        MLflowUI["MLflow UI"]:::external
    end

    %% Connections
    Conda -->|commits code| Build
    Req --> Build
    Build -->|build image| Dockerf
    Dockerf -->|push image| ECR
    RunCloud --> Dockerf
    Run --> Dockerf
    RunCloud --> ECR
    Airflow -->|pull image| ECR
    Airflow --> Data
    Airflow --> Features
    Airflow --> Train
    Airflow --> Predict
    Athena --> Data
    Data -->|reads raw data| S3
    Data --> SQL
    SQL --> Data
    Features -->|writes features| S3
    Train -->|log metrics & models| MetadataDB
    Train -->|store artifacts| S3
    Predict -->|retrieve model| MetadataDB
    Predict -->|write preds| S3
    MLflowUI --> MetadataDB
    MLflowUI --> S3
    MLproject --> Airflow
    Main --> Airflow
    Utils --> Data
    Utils --> Features
    Utils --> Train
    Utils --> Predict
    Example --> Main
    RunTest --> Main
    TestPy --> Main
    Setup --> Main

    %% Click Events
    click Conda "https://github.com/hdegen/mlops-end-to-end/blob/main/src/conda.yaml"
    click Req "https://github.com/hdegen/mlops-end-to-end/blob/main/requirements.txt"
    click Build "https://github.com/hdegen/mlops-end-to-end/blob/main/build_docker.sh"
    click Run "https://github.com/hdegen/mlops-end-to-end/blob/main/run_docker.sh"
    click RunCloud "https://github.com/hdegen/mlops-end-to-end/blob/main/run_docker_cloud.sh"
    click Dockerf "https://github.com/hdegen/mlops-end-to-end/tree/main/Dockerfile"
    click Airflow "https://github.com/hdegen/mlops-end-to-end/blob/main/dag.py"
    click Data "https://github.com/hdegen/mlops-end-to-end/blob/main/src/data/make_data.py"
    click SQL "https://github.com/hdegen/mlops-end-to-end/blob/main/src/data/query.sql"
    click Features "https://github.com/hdegen/mlops-end-to-end/blob/main/src/features/build_features.py"
    click Train "https://github.com/hdegen/mlops-end-to-end/blob/main/src/models/train_model.py"
    click Predict "https://github.com/hdegen/mlops-end-to-end/blob/main/src/models/predict.py"
    click MLproject "https://github.com/hdegen/mlops-end-to-end/tree/main/src/MLproject"
    click Main "https://github.com/hdegen/mlops-end-to-end/blob/main/src/main.py"
    click Utils "https://github.com/hdegen/mlops-end-to-end/blob/main/src/lib/utils.py"
    click Example "https://github.com/hdegen/mlops-end-to-end/blob/main/src/examples/example.sh"
    click RunTest "https://github.com/hdegen/mlops-end-to-end/blob/main/src/run_test.sh"
    click Setup "https://github.com/hdegen/mlops-end-to-end/blob/main/src/setup.py"
    click TestPy "https://github.com/hdegen/mlops-end-to-end/blob/main/src/test.py"

    %% Styles
    classDef compute fill:#D0E6FF,stroke:#0366d6;
    classDef storage fill:#FFE6CC,stroke:#FF8C00;
    classDef ci fill:#DFFFE0,stroke:#28a745;
    classDef external fill:#F0F0F0,stroke:#666666;
    classDef code fill:#F5E6FF,stroke:#6f42c1;
```
