How michaelneale/mesh-llm works
1. Overall shape
- This is a Rust-first monorepo for a distributed LLM runtime and mesh network, with a small TypeScript web UI and a set of packaging, deployment, and evaluation assets.
- The core product lives in mesh-llm/; everything else supports building, shipping, testing, or operating it.
2. Primary runtime boundary
- The main executable is the Rust crate in mesh-llm/.
- It serves two major roles: a local control plane/API for one node, and a mesh-aware coordinator that can join other nodes, route requests, and manage distributed inference.
3. User-facing entry points
- The CLI under mesh-llm/src/cli is the main operator interface for install/run/join/download/status/model management commands.
- The HTTP API under mesh-llm/src/api exposes the local management and OpenAI-compatible serving surface.
- The web console in mesh-llm/ui consumes the API, especially status/event endpoints, to visualize topology and runtime state.
4. Inference and serving path
- mesh-llm/src/inference contains the orchestration logic for launching and coordinating model execution.
- The repository is built around llama.cpp-based backends; the README and scripts indicate a lightly forked llama.cpp with RPC patches for distributed operation.
- Local full-model serving and distributed split serving are treated as different paths, selected based on model fit and node capability.
5. Distributed model execution
- mesh-llm/src/runtime and mesh-llm/src/network handle the distributed side: discovery, peer selection, proxying, tunnels, and request routing across nodes.
- The architecture distinguishes two important distributed modes:
- Pipeline parallelism for dense models too large for one node.
- Expert sharding for MoE models, where experts are split across nodes with a local standalone server per node.
- Network latency matters differently depending on mode: the system avoids network traffic for the steady-state MoE path and only uses RPC heavily when a model must be physically split.
6. Model management subsystem
- mesh-llm/src/models owns catalog lookup, local inventory, GGUF handling, resolution, searching, warnings, and topology-related model decisions.
- The repo ships with a curated model catalog in mesh-llm/src/models/catalog.json.
- Model storage is Hugging Face-centric, with support for local GGUF paths and cached split artifacts for MoE layouts.
7. Mesh communication and routing
- mesh-llm/src/network and mesh-llm/src/mesh are the key boundaries for peer-to-peer behavior.
- These modules cover affinity/peer selection, routing, proxying, tunnel setup, and message rewriting so requests can move between nodes without exposing the distributed complexity to clients.
- proto/node.proto and proto/plugin.proto define the cross-process contracts used by the mesh and plugin layers.
8. Plugins and blackboard
- There is a separate plugin subsystem in mesh-llm/src/plugin plus a compiled plugin crate under mesh-llm/plugin.
- The built-in “blackboard” feature sits on top of this and provides a shared ephemeral message feed for agents and users.
- It can run as a normal mesh feature or as an MCP server over stdio, making it an integration boundary for agent tooling rather than just model serving.
9. Protocol and compatibility layer
- mesh-llm/src/protocol handles conversion and versioned protocol support.
- The README makes clear that the public API is OpenAI-compatible, so external clients can talk to localhost:9337/v1 without mesh-specific knowledge.
- That compatibility layer is also what enables launcher integrations and agent tools to use the mesh as a drop-in backend.
10. Runtime and system management
- mesh-llm/src/system covers hardware detection, benchmarking, and auto-update behavior.
- mesh-llm/src/runtime includes local runtime control and proxy-facing logic for starting/stopping models on a node.
- The architecture treats local node lifecycle as separate from mesh-wide coordination; stage-one load/unload is intentionally node-local.
11. Frontend console
- mesh-llm/ui is a separate Vite/TypeScript app with its own package lock and Tailwind-based component set.
- It is not the control plane; it is a visualization and interaction layer for status, model selection, and chat.
- The console is served alongside the Rust backend and depends on backend APIs rather than maintaining independent state.
12. Deployment and distribution
- install.sh, Justfile, scripts/, and .github/workflows/ form the build/release pipeline.
- dist/ contains service definitions for macOS and Linux background installs.
- fly/ and relay/ indicate deployment support for hosted/public mesh infrastructure, likely used for discovery or transport relaying.
13. Testing, evaluation, and benchmarking
- tests/ plus shell-based test scripts verify blackboard and compatibility behavior.
- evals/ contains scenario-driven evaluation harnesses for agent workflows and latency benchmarks.
- benchmarks/ includes native performance probes for different GPU backends, reflecting the repo’s emphasis on hardware-sensitive behavior.
14. Important architectural boundary to capture in a diagram
- The main split is between:
- Control/UI/CLI: user commands, status, model management, web console.
- Mesh/runtime: discovery, routing, peer selection, tunnels, load balancing.
- Inference backends: local llama.cpp-based servers, RPC workers, and distributed splits.
- The system is designed so a client talks to one local API, while the backend transparently decides whether the model runs locally, is pipelined across peers, or is sharded by experts.
Main entry points: Cargo.toml mesh-llm · api/mod.rs HTTP API · network/mod.rs Mesh network · runtime/mod.rs Runtime
Components 19
Core runtime
Rust crate that serves the API, CLI, mesh, and inference orchestration.
- mesh-llm · Primary executable crate coordinating local serving and mesh-aware operation.
mesh-llm/Cargo.toml - CLI · Install, run, join, download, status, and model-management entry points.
mesh-llm/src/cli/mod.rs - HTTP API · OpenAI-compatible serving surface plus local management and status routes.
mesh-llm/src/api/mod.rs - Inference · Coordinates model startup and execution strategy selection.
mesh-llm/src/inference/mod.rs - Runtime · Local node control, discovery, and proxy-facing runtime state.
mesh-llm/src/runtime/mod.rs - Mesh network · Peer selection, routing, proxying, tunnels, and message rewriting.
mesh-llm/src/network/mod.rs - Models · Catalog, inventory, GGUF handling, topology, and model decisions.
mesh-llm/src/models/mod.rs - Protocol · Versioned protocol conversion and public API compatibility support.
mesh-llm/src/protocol/mod.rs - System · Hardware detection, benchmarking, and autoupdate support.
mesh-llm/src/system/mod.rs - Mesh core · Top-level mesh coordination and cross-node behavior boundary.
mesh-llm/src/mesh/mod.rs - Protos · Cross-process contracts used by mesh and plugin layers.
mesh-llm/proto/node.proto
Extensions
Plugin and blackboard integration boundaries.
- Plugins · Plugin subsystem and built-in plugin integrations.
mesh-llm/src/plugins/mod.rs - Plugin runtime · Transport, MCP, config, and support for plugin execution.
mesh-llm/src/plugin/mod.rs - Blackboard · Ephemeral shared message feed exposed as mesh feature and MCP server.
mesh-llm/src/plugins/blackboard/mod.rs
User interface
TypeScript console for monitoring and interaction.
- Web console · Vite-based frontend for topology, status, and chat interaction.
mesh-llm/ui/src/App.tsx
Build and deploy
Packaging, release, relay, eval, and benchmark assets.
- Release ops · Scripts, CI, install, and packaging paths for shipping the system.
scripts/build-release.sh - Evals · Scenario-driven evaluation, latency measurement, and compatibility smoke tests.
evals/README.md - Benchmarks · GPU and memory benchmark sources used to validate hardware-sensitive behavior.
benchmarks/membench-fingerprint.cu - Relay · Relay deployment support for public mesh connectivity.
relay/README.md
Connections 26
- mesh-llm → CLI: exposes · The main binary wires command-line entry points into the runtime.
- mesh-llm → HTTP API: serves · The crate hosts the HTTP control and OpenAI-compatible API.
- mesh-llm → Inference: orchestrates · The executable coordinates model launch and execution planning.
- mesh-llm → Runtime: manages · Node-local lifecycle and proxy control live in runtime code.
- mesh-llm → Mesh network: routes · Distributed request routing and peer communication are handled here.
- mesh-llm → Models: resolves · Model lookup and local inventory decisions feed serving choices.
- mesh-llm → Protocol: translates · Compatibility and versioned protocol conversion sit on the serving boundary.
- mesh-llm → System: detects · Hardware and update logic inform runtime decisions.
- mesh-llm → Mesh core: coordinates · Mesh-specific behavior is separated from local control concerns.
- HTTP API → Web console: consumed by · The console uses backend status, events, and chat endpoints.
- HTTP API → Protocol: compatible with · The HTTP surface supports OpenAI-style clients.
- CLI → Runtime: controls · Operator commands start, stop, and inspect the local node.
- Inference → Models: selects · Serving strategy depends on model fit, topology, and capabilities.
- Inference → Runtime: launches via · Model execution is started and supervised through node runtime.
- Runtime → Mesh network: uses · Local runtime delegates peer discovery and proxy behavior to the mesh layer.
- Mesh network → Mesh core: implements · Peer routing and request rewriting are the mesh boundary.
- Plugins → Plugin runtime: loads · Built-in and external plugins run through the plugin bridge.
- Plugins → Blackboard: includes · The blackboard feature is a first-class plugin-backed integration.
- Blackboard → HTTP API: exposed by · Blackboard functionality is available through API routes and MCP.
- Plugin runtime → Protos: contracts · Plugin execution relies on shared protobuf interfaces.
- Mesh network → Protos: contracts · Cross-process mesh communication is defined by protobuf contracts.
- Release ops → mesh-llm: packages · Build and release scripts assemble the Rust service.
- Release ops → Web console: bundles · Release tooling also builds the frontend console.
- Evals → HTTP API: verifies · Evals and smoke tests exercise compatibility and agent workflows.
- Benchmarks → System: measures · Native probes validate hardware and backend performance assumptions.
- Relay → Mesh network: supports · Hosted relay infrastructure extends mesh connectivity.