# google-deepmind/xmanager architecture

> Architecture diagram of the GitHub repository google-deepmind/xmanager, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-deepmind/xmanager
- Repository: https://github.com/google-deepmind/xmanager

## Overview

##### Project type and purpose
- XManager is a Python-based framework (library/CLI) for packaging, launching and tracking machine learning experiments.
- It is not a web app but a layered SDK + CLI tool that can run jobs locally (via subprocess or Docker), on Kubernetes, or on GCP (Vertex AI) and record metadata in a database.

##### File structure
- Top-level: README, setup scripts, examples, docs, CLI entrypoint in `xmanager/cli/cli.py`
- Core API: `xmanager/xm/` – primitives for experiments, jobs, executables, packagers, metadata context
- Executors & specs:
- Executable specifications (binary, PythonContainer, Bazel targets) in core + `xmanager/xm_local/packaging`
- Executors (Local, Kubernetes, Vertex) in `xmanager/xm_local` and cloud implementations in `xmanager/cloud`
- Storage: local SQLite with Alembic migrations under `xmanager/xm_local/storage/`
- Contrib & integrations: GCS, ParameterController, TensorBoard, TPU, Vizier under `xmanager/contrib/` and `xmanager/vizier/`
- Generated protocol buffers for event streams and CLI→API contract under `xmanager/generated/`

##### Insights from README
- Core flow: CLI → Python launch script imports `xmanager.xm` and `xmanager.xm_local` → define experiment context → package executables → add jobs → wait/track results.
- Executable specs and executors are pluggable; each executor has its own packaging logic.

##### Components
- CLI (xmanager launch)
- Core API layer (`xmanager.xm`)
- Executable Specs (Binary, PythonContainer, BazelContainer, etc.)
- Executors layer
– Local executor (`xm_local.Local`, Docker adapter)
– Kubernetes executor (`xm_local.Kubernetes`)
– Vertex executor (`cloud/vertex.py`, Cloud Build)
- Packaging / Build (Docker daemon, Bazel, Cloud Build)
- Metadata Storage (SQLite + Alembic migrations)
- Contrib/Extensions (GCS, TensorBoard, Vizier integration)
- Examples & Tutorials (consumers of the API)
- External services (Docker daemon, Kubernetes cluster, GCP APIs: Container Registry, Vertex AI, Cloud Build, GCS)

1. User invokes CLI → CLI parses & invokes Core API.
2. Core API → Experiment Context → calls Packaging sub-system with Executable Spec + Executor Spec.
3. Packaging → interacts with Docker daemon / Bazel / Cloud Build to produce container images or binaries.
4. Core API → submits Job definitions to the chosen Executor impl.
5. Executor (Local / Kubernetes / Vertex) → launches the container or binary (via subprocess, kubectl API, or GCP Vertex AI API).
6. Executor → streams logs and returns job status.
7. Core API → writes experiment & job metadata to SQLite DB via Storage layer.
8. Contrib modules subscribe to metadata (e.g., TensorBoard integration, Parameter Controller, Vizier) to fetch or push additional info.

c. Architectural patterns & principles
- Layered architecture: CLI → Core API → Packaging → Executor → Infrastructure
- Plugin pattern: Executable Specs and Executors registerable, backend-agnostic APIs
- Separation of concerns: core logic in `xmanager.xm`, implementations in `xm_local` and `cloud`, extensions in `contrib` and `vizier`
- Use of protobufs for cross-module contracts and event streams

d. Key technologies/frameworks
- Python 3.9+, setuptools/`setup.py`
- Docker & Docker SDK (for local container packaging)
- Bazel (optional backend)
- GCP: Cloud Build API, Container Registry, Vertex AI API, GCS
- SQLite + Alembic for metadata storage
- Protobufs (`xmanager/generated/`) for CLI ↔ API communication

##### Elements
- Blue for core library modules
- Green for backend implementations
- Orange for external services
- Gray for storage/persistence

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Examples & Tutorials"
        Examples["Examples & Tutorials"]:::example
    end

    subgraph Presentation
        CLI["CLI (xmanager launch)"]:::frontend
    end

    subgraph "Core API Layer"
        Core["Core API (xmanager.xm)"]:::core
        Specs["Executable Specs"]:::core
    end

    subgraph "Packaging & Build"
        DockerPkg["Docker Packaging"]:::backend
        BazelPkg["Bazel Packaging"]:::backend
        CloudBuild["Cloud Build Packaging"]:::backend
    end

    subgraph Executors
        LocalExec["Local Executor"]:::backend
        K8sExec["Kubernetes Executor"]:::backend
        VertexExec["Vertex AI Executor"]:::backend
    end

    subgraph Infrastructure
        DockerDaem["Docker Daemon"]:::external
        K8sCluster["Kubernetes Cluster"]:::external
        GCP["GCP APIs (Container Registry, Vertex AI, Cloud Build, GCS)"]:::external
    end

    subgraph Storage
        DB["SQLite + Alembic"]:::storage
    end

    subgraph Integrations
        GCS["GCS Integration"]:::contrib
        TB["TensorBoard Integration"]:::contrib
        PC["ParameterController"]:::contrib
        TPU["TPU Support"]:::contrib
    end

    subgraph Vizier
        Vizier["Vizier Integration"]:::contrib
    end

    Examples -->|"uses"| CLI
    CLI -->|"invokes"| Core
    Core -->|"defines specs"| Specs
    Core -->|"calls packaging"| DockerPkg
    Core -->|"calls packaging"| BazelPkg
    Core -->|"calls packaging"| CloudBuild
    DockerPkg -->|"Docker API"| DockerDaem
    BazelPkg -->|"Bazel CLI"| GCP
    CloudBuild -->|"Cloud Build API"| GCP
    Core -->|"submits jobs"| LocalExec
    Core -->|"submits jobs"| K8sExec
    Core -->|"submits jobs"| VertexExec
    LocalExec -->|"subprocess / Docker SDK"| DockerDaem
    K8sExec -->|"kubectl API"| K8sCluster
    VertexExec -->|"Vertex AI API"| GCP
    LocalExec -->|"streams logs/status"| Core
    K8sExec -->|"streams logs/status"| Core
    VertexExec -->|"streams logs/status"| Core
    Core -->|"writes metadata"| DB
    GCS -->|"reads/writes metadata"| DB
    TB -->|"reads metadata"| DB
    PC -->|"reads/writes metadata"| DB
    TPU -->|"reads metadata"| DB
    Vizier -->|"reads/writes metadata"| DB
    Vizier -->|"Vertex AI API"| GCP

    click Examples "https://github.com/google-deepmind/xmanager/tree/main/examples/"
    click CLI "https://github.com/google-deepmind/xmanager/blob/main/xmanager/cli/cli.py"
    click Core "https://github.com/google-deepmind/xmanager/tree/main/xmanager/xm/"
    click Specs "https://github.com/google-deepmind/xmanager/blob/main/xmanager/xm/executables.py"
    click DockerPkg "https://github.com/google-deepmind/xmanager/blob/main/xmanager/xm_local/packaging/local.py"
    click BazelPkg "https://github.com/google-deepmind/xmanager/blob/main/xmanager/xm_local/packaging/bazel_tools.py"
    click CloudBuild "https://github.com/google-deepmind/xmanager/blob/main/xmanager/cloud/cloud_build.py"
    click LocalExec "https://github.com/google-deepmind/xmanager/blob/main/xmanager/xm_local/executors.py"
    click K8sExec "https://github.com/google-deepmind/xmanager/blob/main/xmanager/cloud/kubernetes.py"
    click VertexExec "https://github.com/google-deepmind/xmanager/blob/main/xmanager/cloud/vertex.py"
    click DB "https://github.com/google-deepmind/xmanager/tree/main/xmanager/xm_local/storage/"
    click GCS "https://github.com/google-deepmind/xmanager/blob/main/xmanager/contrib/gcs.py"
    click TB "https://github.com/google-deepmind/xmanager/blob/main/xmanager/contrib/tensorboard.py"
    click PC "https://github.com/google-deepmind/xmanager/blob/main/xmanager/contrib/parameter_controller.py"
    click TPU "https://github.com/google-deepmind/xmanager/blob/main/xmanager/contrib/tpu.py"
    click Vizier "https://github.com/google-deepmind/xmanager/tree/main/xmanager/vizier/vizier_cloud/"

    classDef frontend fill:#e3f2fd,stroke:#2196f3,color:#0d47a1
    classDef core fill:#e8eaf6,stroke:#3f51b5,color:#1a237e
    classDef backend fill:#e8f5e9,stroke:#4caf50,color:#1b5e20
    classDef external fill:#fff3e0,stroke:#ff9800,color:#e65100
    classDef storage fill:#f5f5f5,stroke:#9e9e9e,color:#424242
    classDef contrib fill:#f3e5f5,stroke:#ce93d8,color:#4a148c
    classDef example fill:#fffde7,stroke:#fbc02d,color:#f57f17
```
