How google-deepmind/xmanager works
Project type and purpose
• XManager is a Python-based framework (library/CLI) for packaging, launching and tracking machine learning experiments.
• It is not a web app but a layered SDK + CLI tool that can run jobs locally (via subprocess or Docker), on Kubernetes, or on GCP (Vertex AI) and record metadata in a database.
File structure
• Top-level: README, setup scripts, examples, docs, CLI entrypoint in xmanager/cli/cli.py
• Core API: xmanager/xm/ – primitives for experiments, jobs, executables, packagers, metadata context
• Executors & specs:
• Executable specifications (binary, PythonContainer, Bazel targets) in core + xmanager/xm_local/packaging
• Executors (Local, Kubernetes, Vertex) in xmanager/xm_local and cloud implementations in xmanager/cloud
• Storage: local SQLite with Alembic migrations under xmanager/xm_local/storage/
• Contrib & integrations: GCS, ParameterController, TensorBoard, TPU, Vizier under xmanager/contrib/ and xmanager/vizier/
• Generated protocol buffers for event streams and CLI→API contract under xmanager/generated/
Insights from README
• Core flow: CLI → Python launch script imports xmanager.xm and xmanager.xm_local → define experiment context → package executables → add jobs → wait/track results.
• Executable specs and executors are pluggable; each executor has its own packaging logic.
Components
• CLI (xmanager launch)
• Core API layer (xmanager.xm)
• Executable Specs (Binary, PythonContainer, BazelContainer, etc.)
• Executors layer
– Local executor (xm_local.Local, Docker adapter)
– Kubernetes executor (xm_local.Kubernetes)
– Vertex executor (cloud/vertex.py, Cloud Build)
• Packaging / Build (Docker daemon, Bazel, Cloud Build)
• Metadata Storage (SQLite + Alembic migrations)
• Contrib/Extensions (GCS, TensorBoard, Vizier integration)
• Examples & Tutorials (consumers of the API)
• External services (Docker daemon, Kubernetes cluster, GCP APIs: Container Registry, Vertex AI, Cloud Build, GCS)
1. User invokes CLI → CLI parses & invokes Core API.
2. Core API → Experiment Context → calls Packaging sub-system with Executable Spec + Executor Spec.
3. Packaging → interacts with Docker daemon / Bazel / Cloud Build to produce container images or binaries.
4. Core API → submits Job definitions to the chosen Executor impl.
5. Executor (Local / Kubernetes / Vertex) → launches the container or binary (via subprocess, kubectl API, or GCP Vertex AI API).
6. Executor → streams logs and returns job status.
7. Core API → writes experiment & job metadata to SQLite DB via Storage layer.
8. Contrib modules subscribe to metadata (e.g., TensorBoard integration, Parameter Controller, Vizier) to fetch or push additional info.
c. Architectural patterns & principles
• Layered architecture: CLI → Core API → Packaging → Executor → Infrastructure
• Plugin pattern: Executable Specs and Executors registerable, backend-agnostic APIs
• Separation of concerns: core logic in xmanager.xm, implementations in xm_local and cloud, extensions in contrib and vizier
• Use of protobufs for cross-module contracts and event streams
d. Key technologies/frameworks
• Python 3.9+, setuptools/setup.py
• Docker & Docker SDK (for local container packaging)
• Bazel (optional backend)
• GCP: Cloud Build API, Container Registry, Vertex AI API, GCS
• SQLite + Alembic for metadata storage
• Protobufs (xmanager/generated/) for CLI ↔ API communication
Elements
• Blue for core library modules
• Green for backend implementations
• Orange for external services
• Gray for storage/persistence