# statmike/vertex-ai-mlops architecture

> Architecture diagram of the GitHub repository statmike/vertex-ai-mlops, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/statmike/vertex-ai-mlops
- Repository: https://github.com/statmike/vertex-ai-mlops

## Overview

This repository is a large, multifaceted monorepo focused on demonstrating end‐to‐end machine learning operations using Google Cloud’s Vertex AI and other complementary tools. The repository is structured as a collection of workflow demonstrations, examples and tools for MLOps, covering data ingestion, model training with multiple frameworks (e.g., TensorFlow, scikit-learn, PyTorch, R, XGBoost), deployment (including custom training jobs and pipelines), and even applied topics (forecasting, GenAI, autoencoders, etc.). The readme further clarifies that this is an evolving repository geared toward MLOps with a strong focus on Vertex AI, showing how to orchestrate services that perform the heavy lifting on GCP (e.g., training on Vertex AI, running pipelines, monitoring via BigQuery ML and custom endpoints).

##### Main Components
- Data Sources: Create a component for “Data Sources” (e.g., BigQuery tables and various data ingestion methods) that acts as the “input” layer.
- Model Training & Pipelines: Group the folders that demonstrate training workflows – such as “02 - Vertex AI AutoML”, “03 - BigQuery ML (BQML)”, “04 - scikit-learn”, “05 - TensorFlow”, “06 - XGBoost”, “07 - PyTorch”, and “08 - R”. Each can be a node or a set of nodes showing specialized training approaches (AutoML vs. custom training jobs vs. pipelines).
- Applied Workflows: Highlight the “Applied” folders (e.g., Applied Autoencoders, Applied Forecasting, Applied GenAI, etc.) which represent end-to-end workflow examples that integrate data, training, serving, and post-processing.
- MLOps & Deployment: Identify a group of components (folders such as “MLOps”, “Framework Workflows”, and “Dev”) that cover Experiment Tracking, Feature Store, Model Evaluation/Monitoring, and Serving (including endpoints and pipelines). Display these as the orchestration/serving layer.
- External & Integration Services: Include nodes for Pre-Trained APIs and additional integrations (e.g., Document AI, Cloud Translation, Cloud Vision, etc.) as referenced in the overview.

##### Relationships and Interactions
- “Setup” node feeding into all subsequent layers as the initialization process for environments and API setups.
- Interconnections between the “MLOps” layer and the “Monitoring/Experiment Tracking” group, indicating feedback loops (for example, a model monitoring component might feed back insights for retraining).

##### Architectural Patterns and Design Principles
- Separation between exploratory work (notebooks and ad-hoc scripts) and production-ready workflows (MLOps and pipelines), which is akin to evolving from “explore” to “develop” to “deploy.”
- Use of client-side orchestration (via Jupyter notebooks) that offloads heavy compute to cloud-managed services (Vertex AI, BigQuery, etc.).
- Integration with various frameworks to underline that the repository supports multiple paradigms (AutoML vs. custom containerized training) and languages (Python, R, etc.).

##### Technologies, Frameworks, and Libraries
- Clearly label nodes with the associated technologies. For instance, mark “Vertex AI” as the central training and orchestration service, include “BigQuery” for data management and ML (BQML), and annotate specific nodes as “TensorFlow,” “scikit-learn,” “PyTorch,” etc.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Environment Layer
    ENV["Environment & Configuration"]:::env
    click ENV "https://github.com/statmike/vertex-ai-mlops/tree/main/00 - Setup"

    %% Data Sources (Database shaped)
    DS(("Data Sources")):::datasource
    click DS "https://github.com/statmike/vertex-ai-mlops/tree/main/01 - Data Sources"

    ENV -->|"feeds"| DS

    %% Model Training & Pipelines Subgraph
    subgraph "Model Training & Pipelines" 
        direction TB
        T1["Vertex AI AutoML"]:::training
        click T1 "https://github.com/statmike/vertex-ai-mlops/tree/main/02 - Vertex AI AutoML"
        
        T2["BigQuery ML (BQML)"]:::training
        click T2 "https://github.com/statmike/vertex-ai-mlops/tree/main/03 - BigQuery ML (BQML)"
        
        T3["scikit-learn"]:::training
        click T3 "https://github.com/statmike/vertex-ai-mlops/tree/main/04 - scikit-learn"
        
        T4["TensorFlow"]:::training
        click T4 "https://github.com/statmike/vertex-ai-mlops/tree/main/05 - TensorFlow"
        
        T5["XGBoost"]:::training
        click T5 "https://github.com/statmike/vertex-ai-mlops/tree/main/06 - XGBoost"
        
        T6["PyTorch"]:::training
        click T6 "https://github.com/statmike/vertex-ai-mlops/tree/main/07 - PyTorch"
        
        T7["R"]:::training
        click T7 "https://github.com/statmike/vertex-ai-mlops/tree/main/08 - R"
        
        %% Dummy aggregator for training outputs
        TA["Training Outputs"]:::training
    end

    %% Data Sources feeds each training component
    DS --> T1
    DS --> T2
    DS --> T3
    DS --> T4
    DS --> T5
    DS --> T6
    DS --> T7

    %% Each training node outputs to the Training Outputs aggregator
    T1 --> TA
    T2 --> TA
    T3 --> TA
    T4 --> TA
    T5 --> TA
    T6 --> TA
    T7 --> TA

    %% Applied Workflows Subgraph
    subgraph "Applied Workflows"
        direction TB
        A1["Applied Autoencoders"]:::applied
        click A1 "https://github.com/statmike/vertex-ai-mlops/tree/main/Applied Autoencoders"
        
        A2["Applied Dimensionality Reduction"]:::applied
        click A2 "https://github.com/statmike/vertex-ai-mlops/tree/main/Applied Dimensionality Reduction"
        
        A3["Applied Forecasting"]:::applied
        click A3 "https://github.com/statmike/vertex-ai-mlops/tree/main/Applied Forecasting"
        
        A4["Applied GenAI"]:::applied
        click A4 "https://github.com/statmike/vertex-ai-mlops/tree/main/Applied GenAI"
        
        A5["Applied Geospatial"]:::applied
        click A5 "https://github.com/statmike/vertex-ai-mlops/tree/main/Applied Geospatial"
        
        A6["Applied ML"]:::applied
        click A6 "https://github.com/statmike/vertex-ai-mlops/tree/main/Applied ML"
        
        A7["Applied Optimization"]:::applied
        click A7 "https://github.com/statmike/vertex-ai-mlops/tree/main/Applied Optimization"
        
        A8["Applied Pre-Trained APIs"]:::applied
        click A8 "https://github.com/statmike/vertex-ai-mlops/tree/main/Applied Pre-Trained APIs"
        
        A9["Applied Recommender Systems"]:::applied
        click A9 "https://github.com/statmike/vertex-ai-mlops/tree/main/Applied Recommender Systems"
        
        A10["Applied Reinforcement Learning"]:::applied
        click A10 "https://github.com/statmike/vertex-ai-mlops/tree/main/Applied Reinforcement Learning"
        
        A11["Applied Similarity Learning"]:::applied
        click A11 "https://github.com/statmike/vertex-ai-mlops/tree/main/Applied Similarity Learning"
        
        %% Dummy aggregator for applied workflows
        AA["Applied Aggregator"]:::applied
    end

    %% MLOps & Deployment Subgraph
    subgraph "MLOps & Deployment"
        direction TB
        MA["Deployment & MLOps Aggregator"]:::mlops
        M1["MLOps"]:::mlops
        click M1 "https://github.com/statmike/vertex-ai-mlops/tree/main/MLOps"
        M2["Framework Workflows"]:::mlops
        click M2 "https://github.com/statmike/vertex-ai-mlops/tree/main/Framework Workflows"
        M3["Dev"]:::mlops
        click M3 "https://github.com/statmike/vertex-ai-mlops/tree/main/Dev"
    end

    %% External & Integration Services
    E1["Document AI Integration"]:::external
    click E1 "https://github.com/statmike/vertex-ai-mlops/tree/main/Working With/Document AI"

    %% Data Flow Connections
    %% Training Aggregator connections
    TA -->|"toApplied"| AA
    TA -->|"directDeploy"| MA

    %% Applied Workflows feed into Applied Aggregator
    A1 --> AA
    A2 --> AA
    A3 --> AA
    A4 --> AA
    A5 --> AA
    A6 --> AA
    A7 --> AA
    A8 --> AA
    A9 --> AA
    A10 --> AA
    A11 --> AA

    %% Applied Aggregator feeds into MLOps Aggregator
    AA --> MA

    %% MLOps & Deployment internal sequence
    MA --> M1
    M1 --> M2
    M2 --> M3

    %% MLOps Aggregator deploys to External Integration
    M3 --> E1

    %% Feedback loop from deployment back to training outputs
    M3 -->|"feedback"| TA

    %% Styles
    classDef env fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
    classDef datasource fill:#ccffcc,stroke:#008000,stroke-width:2px;
    classDef training fill:#cce5ff,stroke:#0059b3,stroke-width:2px;
    classDef applied fill:#fff2cc,stroke:#e68a00,stroke-width:2px;
    classDef mlops fill:#d9ccff,stroke:#663399,stroke-width:2px;
    classDef external fill:#e6f9ff,stroke:#0099cc,stroke-width:2px;
```
