# paulescu/end-2-end-real-time-ml architecture

> Architecture diagram of the GitHub repository paulescu/end-2-end-real-time-ml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/paulescu/end-2-end-real-time-ml
- Repository: https://github.com/paulescu/end-2-end-real-time-ml

## Overview

##### Main Components
- Dev Environment & Infrastructure:
- "devcontainer" (the pre-configured Docker-based development environment)
- .env configuration (for TurboML credentials)
- Data Ingestion & Feature Engineering:
- The datasets that comprise raw transaction data (as simulated through the script and dataset creation in setup_feature_pipeline.py)
- The feature engineering component (the logic encapsulated in setup_feature_pipeline.py which processes these datasets)
- Machine Learning Model Handling:
- Offline training and online deployment of the model (the setup_model.py script outlines how the model is defined and deployed)
- Model monitoring and comparison components, which are part of the TurboML dashboard and the iterative process described in the README
- Live Data Generation:
- The generate_live_data.py script that simulates a live data stream (representing real-time transaction data)
- External TurboML Platform:
- Represents the TurboML backend which handles model deployment, real-time predictions, monitoring, and connectors (data sources like Kafka or databases in a production scenario)

##### Relationships and Interactions Between Components
- There’s interaction between the setup_feature_pipeline.py (which creates and processes datasets) and the subsequent model training/online deployment via setup_model.py.
- Additionally, indicate the connection between the development environment (devcontainer) which hosts and runs the scripts locally, and the TurboML platform that handles the heavy lifting (deployment, monitoring, and scaling).

##### Architectural Patterns and Design Principles
- Modularity: Each major step (pipeline setup, model deployment, live predictions, monitoring, etc.) is handled by distinct Python scripts.
- Separation of Concerns: The project separates configuration (config.py), feature pipeline (setup_feature_pipeline.py), model deployment (setup_model.py), and live data simulation (generate_live_data.py).
- Containerization: Usage of a devcontainer ensures a reproducible development environment.
- Integration with a cloud-based service (TurboML) to manage infrastructure, serving as the backbone for real-time predictions and model monitoring.

##### Relevant Technologies and Tools
- Docker (via devcontainer) and VSCode (for the development environment)
- Python (all scripts and configuration files)
- TurboML platform (for model deployment, monitoring, and API endpoints)
- Makefile (to streamline running commands like “make feature-pipeline”, “make model”, and “make live-data”)
- Environment configuration (.env) to manage sensitive credentials

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Development Environment"
        DevEnv["Devcontainer Environment"]:::dev
        EnvConfig[".env configuration"]:::dev
        BuildAutomation["Build & Command Automation (Makefile)"]:::dev
        Configuration["Configuration Management (config.py)"]:::dev
    end

    subgraph "Data Ingestion & Feature Engineering"
        FeatureEng["Feature Engineering Module (setup_feature_pipeline.py)"]:::data
    end

    subgraph "Machine Learning Model Handling"
        MLModel["ML Model Training/Deployment (setup_model.py)"]:::model
    end

    subgraph "Live Data Generation"
        LiveGen["Live Prediction Generator (generate_live_data.py)"]:::live
    end

    subgraph "TurboML Platform / Dashboard"
        TurboML["TurboML Platform"]:::external
    end

    %% Relationships
    DevEnv -->|"hosts"| FeatureEng
    BuildAutomation -->|"triggers"| FeatureEng
    Configuration -->|"configures"| FeatureEng
    EnvConfig -->|"supplies env"| DevEnv

    LiveGen -->|"streams data"| FeatureEng
    FeatureEng -->|"processed features"| MLModel
    MLModel -->|"deploys model"| TurboML
    TurboML -->|"feedback"| MLModel

    %% Click Events
    click DevEnv "https://github.com/paulescu/end-2-end-real-time-ml/blob/main/.devcontainer"
    click EnvConfig "https://github.com/paulescu/end-2-end-real-time-ml/blob/main/.env.example"
    click BuildAutomation "https://github.com/paulescu/end-2-end-real-time-ml/tree/main/Makefile"
    click Configuration "https://github.com/paulescu/end-2-end-real-time-ml/blob/main/config.py"
    click FeatureEng "https://github.com/paulescu/end-2-end-real-time-ml/blob/main/setup_feature_pipeline.py"
    click MLModel "https://github.com/paulescu/end-2-end-real-time-ml/blob/main/setup_model.py"
    click LiveGen "https://github.com/paulescu/end-2-end-real-time-ml/blob/main/generate_live_data.py"

    %% Styles
    classDef dev fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef data fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef model fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef live fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:2px;
```
