How googlecloudplatform/ai-on-gke works
Project type and purpose
• This is an open-source “AI on GKE” platform repository. It provides Infrastructure-as-Code (Terraform modules), Kubernetes Helm charts, CI/CD integration, and custom controllers/operators (Go, Python) to deploy and manage AI/ML workloads (JupyterHub, Ray, inference servers, benchmarks) on Google Kubernetes Engine (GKE).
High-level file-structure analysis
• Top-level directories:
• infrastructure & modules – Terraform code for provisioning clusters and GCP resources (GKE clusters, networking, CloudSQL, IAP, custom metrics adapters)
• charts – Helm charts to install GPU/TPU drivers, platform components (gmp-engine) into the cluster
• applications – Terraform “application modules” to stand up JupyterHub and Ray (Kuberay) once the cluster exists
• benchmarks & orchestration – Benchmark definitions, Terraform/Helm configs and scripts to spin up load and measure performance
• tpu-provisioner – A Kubernetes admission-controller/operator implemented in Go to dynamically create and manage TPU resources
• tools – Utility CLI programs, Dockerized quickstarts, custom servers, Python scripts for profiling, load-generation
• best-practices, tutorials-and-examples – Documentation, sample workloads, end-to-end tutorials
• security_test – Policy allowlists for cluster and Helm chart security testing
• CI/CD: cloudbuild_*.yaml, GitHub workflows, Makefile
Core README insights
• Purpose: Provide layered AI/ML support on GKE – from infra orchestration to notebook and distributed-compute frameworks.
• Workflows:
• Bootstrap a GKE cluster via Terraform (infrastructure/)
• Apply platform modules (modules/) – network, private/public clusters, IAP, metrics
• Deploy applications (applications/jupyter, applications/ray) via Terraform
• Install drivers and platform services via Helm (charts/)
• Run benchmarks, profiling and best-practices examples
• Emphasis on multi-team support, GPU/TPU orchestration, extensibility via modules and charts.
Main components and relationships (
A. Control Plane (Your “deployment engine” outside the cluster)
• Terraform CLI & State (backed by GCS)
• GitHub Actions / Cloud Build pipelines (cloudbuild_*.yaml, .github/workflows)
• Makefile / scripts for orchestration
B. GCP Resources
• VPC, Subnets (modules/gcp-network)
• CloudSQL (modules/cloudsql)
• IAP, OAuth clients (modules/iap)
• Kubernetes Engine clusters (modules/gke-*)
C. Kubernetes Cluster
1. Infrastructure Services (installed via Helm or manifests)
– GPU & TPU device plugins and drivers (charts/nvidia-dra-driver-gpu, charts/tpu-dra-driver)
– Custom-metrics Stackdriver Adapter (modules/custom-metrics-stackdriver-adapter)
– Prometheus Adapter (modules/prometheus-adapter)
2. Operators & Controllers
– tpu-provisioner admission controller (tpu-provisioner/)
– KubeRay operator (modules/kuberay-cluster + apps)
– Custom metrics & autoscaling controllers (bundled in modules and charts)
3. Applications & Workloads
– JupyterHub (modules/jupyter + applications/jupyter)
– Ray clusters & Ray AIR notebooks (modules/kuberay, applications/ray)
– Inference Services (Jetstream, Triton, vLLM under benchmarks/inference-server)
– Benchmarks (pods launched via Terraform or Helm under benchmarks/benchmark/tools)
4. Supporting Infrastructure
– Namespaces per user/team (modules/kubernetes-namespace)
– Service Accounts, RBAC (modules/jupyter, modules/iap, security_test allowlists)
– Monitoring & Logging (Prometheus, Grafana dashboards under tools and charts)
D. External Users & Clients
• Data scientists connecting to Jupyter/Ray UI
• Benchmarking clients (locust, dlio, custom HTTP servers) executing against inference endpoints
• CI/CD systems triggering terraform/apply and running tests
Architectural patterns & design principles
• Infrastructure-as-Code (Terraform modules, tfvars, backends)
• Modular design – per-service Terraform modules, per-component Helm charts
• Controller/Operator pattern – tpu-provisioner, KubeRay, custom metrics adapters
• Separation of concerns – infra provisioning vs. platform installation vs. application deployment vs. benchmarking
• GitOps/CI pipelines – GitHub workflows, Cloud Build, Makefile for reproducibility
Technologies & frameworks
• Terraform (infrastructure/, modules/, benchmarks/orchestration)
• Kubernetes & GKE
• Helm charts (charts/)
• Go (tpu-provisioner, CLI tools, benchmark tools)
• Python (JupyterHub, HTTP servers, plot scripts)
• Docker (images for jupyter, Ray, tools)
• Prometheus & Grafana (monitoring templates)
• IAP, GCS, CloudSQL, VPC – GCP managed services