# googlecloudplatform/ai-on-gke architecture

> Architecture diagram of the GitHub repository googlecloudplatform/ai-on-gke, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/googlecloudplatform/ai-on-gke
- Repository: https://github.com/googlecloudplatform/ai-on-gke

## Overview

##### Project type and purpose
- This is an open-source “AI on GKE” platform repository. It provides Infrastructure-as-Code (Terraform modules), Kubernetes Helm charts, CI/CD integration, and custom controllers/operators (Go, Python) to deploy and manage AI/ML workloads (JupyterHub, Ray, inference servers, benchmarks) on Google Kubernetes Engine (GKE).

##### High-level file-structure analysis
- Top-level directories:
- infrastructure & modules – Terraform code for provisioning clusters and GCP resources (GKE clusters, networking, CloudSQL, IAP, custom metrics adapters)
- charts – Helm charts to install GPU/TPU drivers, platform components (gmp-engine) into the cluster
- applications – Terraform “application modules” to stand up JupyterHub and Ray (Kuberay) once the cluster exists
- benchmarks & orchestration – Benchmark definitions, Terraform/Helm configs and scripts to spin up load and measure performance
- tpu-provisioner – A Kubernetes admission-controller/operator implemented in Go to dynamically create and manage TPU resources
- tools – Utility CLI programs, Dockerized quickstarts, custom servers, Python scripts for profiling, load-generation
- best-practices, tutorials-and-examples – Documentation, sample workloads, end-to-end tutorials
- security_test – Policy allowlists for cluster and Helm chart security testing
- CI/CD: cloudbuild_*.yaml, GitHub workflows, Makefile

##### Core README insights
- Purpose: Provide layered AI/ML support on GKE – from infra orchestration to notebook and distributed-compute frameworks.
- Workflows:
- Bootstrap a GKE cluster via Terraform (infrastructure/)
- Apply platform modules (modules/) – network, private/public clusters, IAP, metrics
- Deploy applications (applications/jupyter, applications/ray) via Terraform
- Install drivers and platform services via Helm (charts/)
- Run benchmarks, profiling and best-practices examples
- Emphasis on multi-team support, GPU/TPU orchestration, extensibility via modules and charts.

##### Main components and relationships (
A. Control Plane (Your “deployment engine” outside the cluster)
- Terraform CLI & State (backed by GCS)
- GitHub Actions / Cloud Build pipelines (cloudbuild_*.yaml, .github/workflows)
- Makefile / scripts for orchestration
B. GCP Resources
- VPC, Subnets (modules/gcp-network)
- CloudSQL (modules/cloudsql)
- IAP, OAuth clients (modules/iap)
- Kubernetes Engine clusters (modules/gke-*)
C. Kubernetes Cluster
1. Infrastructure Services (installed via Helm or manifests)
– GPU & TPU device plugins and drivers (charts/nvidia-dra-driver-gpu, charts/tpu-dra-driver)
– Custom-metrics Stackdriver Adapter (modules/custom-metrics-stackdriver-adapter)
– Prometheus Adapter (modules/prometheus-adapter)
2. Operators & Controllers
– tpu-provisioner admission controller (tpu-provisioner/)
– KubeRay operator (modules/kuberay-cluster + apps)
– Custom metrics & autoscaling controllers (bundled in modules and charts)
3. Applications & Workloads
– JupyterHub (modules/jupyter + applications/jupyter)
– Ray clusters & Ray AIR notebooks (modules/kuberay, applications/ray)
– Inference Services (Jetstream, Triton, vLLM under benchmarks/inference-server)
– Benchmarks (pods launched via Terraform or Helm under benchmarks/benchmark/tools)
4. Supporting Infrastructure
– Namespaces per user/team (modules/kubernetes-namespace)
– Service Accounts, RBAC (modules/jupyter, modules/iap, security_test allowlists)
– Monitoring & Logging (Prometheus, Grafana dashboards under tools and charts)
D. External Users & Clients
- Data scientists connecting to Jupyter/Ray UI
- Benchmarking clients (locust, dlio, custom HTTP servers) executing against inference endpoints
- CI/CD systems triggering terraform/apply and running tests

##### Architectural patterns & design principles
- Infrastructure-as-Code (Terraform modules, tfvars, backends)
- Modular design – per-service Terraform modules, per-component Helm charts
- Controller/Operator pattern – tpu-provisioner, KubeRay, custom metrics adapters
- Separation of concerns – infra provisioning vs. platform installation vs. application deployment vs. benchmarking
- GitOps/CI pipelines – GitHub workflows, Cloud Build, Makefile for reproducibility

##### Technologies & frameworks
- Terraform (infrastructure/, modules/, benchmarks/orchestration)
- Kubernetes & GKE
- Helm charts (charts/)
- Go (tpu-provisioner, CLI tools, benchmark tools)
- Python (JupyterHub, HTTP servers, plot scripts)
- Docker (images for jupyter, Ray, tools)
- Prometheus & Grafana (monitoring templates)
- IAP, GCS, CloudSQL, VPC – GCP managed services

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Control Plane
    subgraph "Control Plane" 
        direction TB
        TerraformCLI["Terraform CLI & State"]:::terraform
        GitHubCI["GitHub Actions / Cloud Build"]:::terraform
        Makefile["Makefile Orchestration"]:::terraform
    end

    %% GCP Resources
    subgraph "GCP Resources"
        direction TB
        VPC["VPC & Subnets"]:::gcp
        CloudSQL["CloudSQL"]:::gcp
        IAP["IAP & OAuth Clients"]:::gcp
        subgraph "GKE Clusters"
            direction TB
            GKE_SP["Standard Public Cluster"]:::gcp
            GKE_SPr["Standard Private Cluster"]:::gcp
            GKE_AP["Autopilot Public Cluster"]:::gcp
            GKE_APr["Autopilot Private Cluster"]:::gcp
        end
    end

    %% Kubernetes Cluster
    subgraph "Kubernetes Cluster"
        direction TB

        subgraph "Infrastructure Services"
            direction TB
            GPU["NVIDIA GPU Plugin & Driver"]:::kub
            TPU["TPU Plugin & Driver"]:::kub
            CMSA["Custom Metrics Stackdriver Adapter"]:::kub
            PA["Prometheus Adapter"]:::kub
        end

        subgraph "Operators & Controllers"
            direction TB
            TPUProv["TPU Provisioner Controller"]:::kub
            KROp["KubeRay Operator"]:::kub
            KRMon["KubeRay Monitoring"]:::kub
        end

        subgraph "Applications & Workloads"
            direction TB
            Jupyter["JupyterHub"]:::app
            Ray["Ray Clusters & Notebooks"]:::app
            Jetstream["Jetstream Inference"]:::app
            Triton["Triton Inference"]:::app
            vLLM["vLLM Inference"]:::app
            BenchTools["Benchmark Tools"]:::app
        end

        subgraph "Supporting Infrastructure"
            direction TB
            Namespaces["Kubernetes Namespaces per Team"]:::kub
            DCGM["DCGM Monitoring Proxy"]:::kub
        end
    end

    %% External Clients
    subgraph "External Clients"
        direction TB
        DataSci["Data Scientists"]:::external
        BenchClients["Benchmarking Clients"]:::external
    end

    %% Connections
    TerraformCLI -->|"terraform apply"| VPC
    GitHubCI -->|"trigger terraform"| TerraformCLI
    Makefile -->|"orchestrate"| TerraformCLI
    TerraformCLI -->|"deploy modules"| CloudSQL
    TerraformCLI -->|"deploy modules"| IAP
    TerraformCLI -->|"deploy modules"| GKE_SP
    TerraformCLI -->|"deploy modules"| GKE_SPr
    TerraformCLI -->|"deploy modules"| GKE_AP
    TerraformCLI -->|"deploy modules"| GKE_APr

    GKE_SP -->|"cluster up"| GPU
    GKE_SP -->|"cluster up"| TPU
    GKE_SP -->|"cluster up"| CMSA
    GKE_SP -->|"cluster up"| PA

    GPU -->|"helm install"| GPU
    GPU -->|"node plugin"| Operators & Controllers
    TPUProv -->|"calls Cloud API"| IAP
    GKE_SP -->|"kubectl"| Jupyter
    GKE_SP -->|"kubectl"| Ray
    GKE_SP -->|"kubectl"| Jetstream
    GKE_SP -->|"kubectl"| Triton
    GKE_SP -->|"kubectl"| vLLM
    BenchTools -->|"launch pods"| GKE_SP

    DataSci -->|"HTTP/HTTPS"| Jupyter
    DataSci -->|"HTTP/HTTPS"| Ray
    BenchClients -->|"HTTP"| Jetstream
    BenchClients -->|"HTTP"| Triton
    BenchClients -->|"HTTP"| vLLM

    Jetstream -->|"metrics"| PA
    Triton -->|"metrics"| CMSA
    vLLM -->|"metrics"| CMSA

    Namespaces -->|"create"| Jupyter
    Namespaces -->|"create"| Ray

    %% Click Events
    click TerraformCLI "https://github.com/googlecloudplatform/ai-on-gke/tree/main/infrastructure/"
    click GitHubCI "https://github.com/googlecloudplatform/ai-on-gke/blob/main/cloudbuild_cleanup.yaml"
    click Makefile "https://github.com/googlecloudplatform/ai-on-gke/tree/main/Makefile"
    click VPC "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/gcp-network/"
    click CloudSQL "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/cloudsql/"
    click IAP "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/iap/"
    click GKE_SP "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/gke-standard-public-cluster/"
    click GKE_SPr "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/gke-standard-private-cluster/"
    click GKE_AP "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/gke-autopilot-public-cluster/"
    click GKE_APr "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/gke-autopilot-private-cluster/"
    click GPU "https://github.com/googlecloudplatform/ai-on-gke/tree/main/charts/nvidia-dra-driver-gpu/"
    click TPU "https://github.com/googlecloudplatform/ai-on-gke/tree/main/charts/tpu-dra-driver/"
    click CMSA "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/custom-metrics-stackdriver-adapter/"
    click PA "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/prometheus-adapter/"
    click TPUProv "https://github.com/googlecloudplatform/ai-on-gke/tree/main/tpu-provisioner/"
    click KROp "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/kuberay-cluster/"
    click KRMon "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/kuberay-monitoring/"
    click Jupyter "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/jupyter/"
    click Ray "https://github.com/googlecloudplatform/ai-on-gke/tree/main/applications/ray/"
    click Jetstream "https://github.com/googlecloudplatform/ai-on-gke/tree/main/benchmarks/inference-server/jetstream/"
    click Triton "https://github.com/googlecloudplatform/ai-on-gke/tree/main/benchmarks/inference-server/triton/"
    click vLLM "https://github.com/googlecloudplatform/ai-on-gke/tree/main/benchmarks/inference-server/vllm/"
    click BenchTools "https://github.com/googlecloudplatform/ai-on-gke/tree/main/benchmarks/benchmark/tools/"
    click Namespaces "https://github.com/googlecloudplatform/ai-on-gke/tree/main/modules/kubernetes-namespace/"
    click DCGM "https://github.com/googlecloudplatform/ai-on-gke/tree/main/tools/dcgm-on-gke/"

    %% Styles
    classDef terraform fill:#D6EAF8,stroke:#3498DB
    classDef gcp fill:#D5F5E3,stroke:#27AE60
    classDef kub fill:#FCF3CF,stroke:#F1C40F
    classDef app fill:#F5EEF8,stroke:#8E44AD
    classDef external fill:#ECECEC,stroke:#7F8C8Dendmodule
```
