How exo-explore/exo works
1. Overall shape
exo is a multi-process, multi-language local AI cluster system. The main runtime is a Python package (src/exo) that acts as the coordinator/control plane and also hosts worker logic. It is complemented by Rust networking/runtime helpers, a Swift macOS app wrapper, and a Svelte dashboard UI.
2. Control plane vs. worker plane
The Python code is split around a clear boundary:
- src/exo/master manages cluster state, placement, and instance lifecycle.
- src/exo/worker runs model execution, loading, batching, and runner supervision.
- src/exo/api exposes external compatibility APIs and command endpoints.
This separation is important for diagrams: the master decides what should run, while workers actually execute inference.
3. Cluster coordination and state
The master subsystem keeps the authoritative view of the cluster: nodes, instances, topology, and placement decisions. Files under src/exo/shared/types, src/exo/shared/topology.py, src/exo/shared/election.py, and src/exo/master/placement.py suggest that cluster state is modeled explicitly and serialized across processes/nodes. The dashboard’s cluster view and /state-style API endpoints are fed from this state.
4. Networking and device discovery
Low-level discovery and swarm membership live in Rust (rust/networking) and are likely consumed through Python bindings in rust/exo_rs. This layer handles device discovery, heartbeats, and cluster connectivity, including the Thunderbolt/RDMA-oriented transport path mentioned in the README. The Python side uses these network events to build a realtime topology and coordinate workers.
5. Inference execution path
Inference flows from API request to runner execution:
- A request arrives through an API adapter (chat_completions, claude, responses, ollama).
- It is mapped into internal task/command types in src/exo/shared/types.
- The worker runner layer (src/exo/worker/runner) executes the request using a selected engine.
- Streaming results are returned to the client, usually over SSE/streaming HTTP.
The architecture is request-driven and asynchronous; instance creation and deletion are commands, while generation is a separate streaming path.
6. Model execution engines
The worker supports at least two major engine families:
- src/exo/worker/engines/mlx for text/LLM inference on Apple Silicon/macOS, including batching, cache, distributed execution, and patches.
- src/exo/worker/engines/image for image generation models, with model-specific adapters for Flux and Qwen.
The README’s macOS GPU support and Linux CPU-only note materially affect deployment diagrams: MLX is the primary backend, while Linux is currently a reduced-capability environment.
7. Model catalog, downloading, and weight mapping
Model lifecycle is split into cataloging, download, and weight application:
- src/exo/shared/models/model_cards.py and the resources/*_model_cards TOML files define known models and presets.
- src/exo/download/* handles Hugging Face resolution, shard downloads, retries, verification, offline mode, and cancellation.
- src/exo/worker/engines/*/weights and shared weight-mapping code convert model-specific checkpoints into runtime-ready tensors.
This is a major architectural boundary: model metadata and storage are separate from the runtime execution graph.
8. Placement, topology, and distributed planning
The cluster is topology-aware. src/exo/master/placement.py, src/exo/master/placement_utils.py, and src/exo/shared/topology.py imply that exo evaluates node memory, connectivity, and latency before deciding how to shard a model. The README’s “preview placements” endpoint reflects this design: placement is computed before instance creation, not hardcoded. Tensor parallelism and pipeline-style distribution are first-class concepts.
9. Shared types and eventing
Most cross-cutting contracts are centralized in src/exo/shared/types and utility modules like src/exo/utils/event_buffer.py, channels.py, reactive.py, and disk_event_log.py. These provide the message/event vocabulary for commands, tasks, worker status, topology updates, and streamed tokens. This is the glue between master, worker, API, and UI.
10. External API compatibility layer
src/exo/api exposes multiple front doors that all map onto the same internal execution model:
- OpenAI Chat Completions
- Anthropic Claude Messages
- OpenAI Responses
- Ollama
This adapter layer is a key diagram boundary: external clients speak different schemas, but they converge on the same instance/task/streaming pipeline.
11. User-facing surfaces
There are two primary user interfaces:
- The Svelte dashboard in dashboard/, which provides cluster, model, download, trace, and chat views.
- The macOS Swift app in app/EXO/, which manages launch, network setup, settings, and uninstall behavior.
The dashboard is the interactive control surface; the app is the native system-integrated shell for macOS.
12. Platform integration and packaging
The repo includes substantial platform-specific support:
- Rust crates for networking/runtime helpers.
- Swift app packaging and macOS network/location setup.
- Nix, PyPI/uv, and PyInstaller/dmg packaging.
- Typing stubs under .typings/ for MLX, mlx-lm, safetensors, and internal modules.
This indicates exo is not a single deployment artifact, but a platformed system with native, Python, and JS components.
13. Benchmarks, tests, and support tooling
bench/, tests/, and the various src/exo/*/tests directories show that performance evaluation, cluster resilience, API compatibility, and distributed correctness are treated as first-class concerns. tools/ contains an auxiliary client/harness package, while tmp/ holds operational scripts and historical experiments.
14. High-level data flow to diagram
The most accurate architecture diagram would show:
external client or dashboard → Python API layer → master/placement/state → worker/runner/engine → model downloads/weights → streamed response back to client.
In parallel, discovery/networking feeds the master with node presence and topology, and the Swift app / Rust layer handles system and transport concerns underneath the Python control plane.
Main entry points: master/main.py Master · runner.py Runner · api/main.py API server · exo/main.py exo entry
Components 20
Python runtime
Coordinator, APIs, worker logic, and shared contracts
- exo entry · Top-level package launcher and process bootstrap
src/exo/main.py - API server · External API front door and command/stream handling
src/exo/api/main.py - API adapters · OpenAI, Anthropic, Ollama, and Responses request translation
src/exo/api/adapters - Master · Authoritative cluster state, lifecycle, and orchestration
src/exo/master/main.py - Placement · Topology-aware model placement and instance allocation
src/exo/master/placement.py - Shared types · Commands, tasks, state, topology, and worker-facing schemas
src/exo/shared/types - Topology · Node presence, connectivity, memory, and election state
src/exo/shared/topology.py - Event bus · Buffered events, channels, reactive state, and disk event log
src/exo/utils/event_buffer.py - Worker · Worker process that hosts model execution and supervision
src/exo/worker/main.py - Runner · Task execution, lifecycle management, diagnostics, and streaming results
src/exo/worker/runner/runner.py - MLX engine · Apple Silicon inference engine with batching, cache, and distributed support
src/exo/worker/engines/mlx - Image engine · Flux and Qwen image generation pipeline and model adapters
src/exo/worker/engines/image - Downloads · Hugging Face resolution, shard download, verification, offline mode, and cancellation
src/exo/download - Model catalog · Known models and presets used by the coordinator and worker
src/exo/shared/models/model_cards.py - Routing · Topic routing and connection message handling between processes
src/exo/routing/router.py
Native layers
Rust networking/runtime helpers and Swift macOS shell
- Rust networking · Discovery, swarm membership, heartbeats, and low-level cluster connectivity
rust/networking/src - Rust bindings · Bindings and helpers exposed to Python for networking/runtime support
rust/exo_rs/src/lib.rs
User surfaces
Dashboard and macOS app
Support tooling
Benchmarks, scripts, and packaging
- Bench/tools · Benchmark harnesses, scripts, and operational utilities
bench
Connections 23
- exo entry → API server: starts · Launches the API/control-plane process
- exo entry → Master: starts · Launches cluster coordinator logic
- exo entry → Worker: starts · Launches worker execution process
- API server → API adapters: maps requests · Normalizes external API schemas into internal commands
- API adapters → Shared types: emits · Converts client payloads into shared commands and tasks
- API server → Routing: routes · Dispatches commands and streams events through internal routing
- Master → Shared types: uses · Reads and writes cluster state and command schemas
- Master → Topology: tracks · Maintains authoritative cluster topology and node state
- Master → Placement: plans · Computes placement before instance creation
- Placement → Model catalog: selects · Uses model metadata and presets to choose layout
- Worker → Runner: supervises · Hosts execution lifecycle and task forwarding
- Runner → MLX engine: executes · Runs text inference and generation on MLX
- Runner → Image engine: executes · Runs image generation workflows
- Runner → Downloads: loads models · Triggers model fetch and verification when needed
- Downloads → Model catalog: resolves · Uses catalog metadata to locate and validate weights
- Rust networking → Rust bindings: exposed via · Networking/runtime helpers are surfaced to Python
- Rust bindings → Topology: feeds · Provides discovery and membership signals to cluster state
- Rust networking → Master: signals · Discovery and heartbeats inform coordinator state
- Dashboard → API server: consumes · Reads state and sends control requests through the API
- Mac app → Rust bindings: uses · Native shell relies on Rust helpers for process and network setup
- Mac app → Dashboard: opens · Can launch or surface the dashboard experience
- Bench/tools → API server: evaluates · Exercises API, streaming, and cluster behavior
- Bench/tools → Runner: measures · Benchmarks worker throughput and latency