# michaelneale/mesh-llm architecture

> Architecture diagram of the GitHub repository michaelneale/mesh-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/michaelneale/mesh-llm
- Repository: https://github.com/michaelneale/mesh-llm

## Overview

1. **Overall shape**
   - This is a Rust-first monorepo for a distributed LLM runtime and mesh network, with a small TypeScript web UI and a set of packaging, deployment, and evaluation assets.
   - The core product lives in `mesh-llm/`; everything else supports building, shipping, testing, or operating it.

2. **Primary runtime boundary**
   - The main executable is the Rust crate in `mesh-llm/`.
   - It serves two major roles: a local control plane/API for one node, and a mesh-aware coordinator that can join other nodes, route requests, and manage distributed inference.

3. **User-facing entry points**
   - The CLI under `mesh-llm/src/cli` is the main operator interface for install/run/join/download/status/model management commands.
   - The HTTP API under `mesh-llm/src/api` exposes the local management and OpenAI-compatible serving surface.
   - The web console in `mesh-llm/ui` consumes the API, especially status/event endpoints, to visualize topology and runtime state.

4. **Inference and serving path**
   - `mesh-llm/src/inference` contains the orchestration logic for launching and coordinating model execution.
   - The repository is built around llama.cpp-based backends; the README and scripts indicate a lightly forked llama.cpp with RPC patches for distributed operation.
   - Local full-model serving and distributed split serving are treated as different paths, selected based on model fit and node capability.

5. **Distributed model execution**
   - `mesh-llm/src/runtime` and `mesh-llm/src/network` handle the distributed side: discovery, peer selection, proxying, tunnels, and request routing across nodes.
   - The architecture distinguishes two important distributed modes:
     - **Pipeline parallelism** for dense models too large for one node.
     - **Expert sharding** for MoE models, where experts are split across nodes with a local standalone server per node.
   - Network latency matters differently depending on mode: the system avoids network traffic for the steady-state MoE path and only uses RPC heavily when a model must be physically split.

6. **Model management subsystem**
   - `mesh-llm/src/models` owns catalog lookup, local inventory, GGUF handling, resolution, searching, warnings, and topology-related model decisions.
   - The repo ships with a curated model catalog in `mesh-llm/src/models/catalog.json`.
   - Model storage is Hugging Face-centric, with support for local GGUF paths and cached split artifacts for MoE layouts.

7. **Mesh communication and routing**
   - `mesh-llm/src/network` and `mesh-llm/src/mesh` are the key boundaries for peer-to-peer behavior.
   - These modules cover affinity/peer selection, routing, proxying, tunnel setup, and message rewriting so requests can move between nodes without exposing the distributed complexity to clients.
   - `proto/node.proto` and `proto/plugin.proto` define the cross-process contracts used by the mesh and plugin layers.

8. **Plugins and blackboard**
   - There is a separate plugin subsystem in `mesh-llm/src/plugin` plus a compiled plugin crate under `mesh-llm/plugin`.
   - The built-in “blackboard” feature sits on top of this and provides a shared ephemeral message feed for agents and users.
   - It can run as a normal mesh feature or as an MCP server over stdio, making it an integration boundary for agent tooling rather than just model serving.

9. **Protocol and compatibility layer**
   - `mesh-llm/src/protocol` handles conversion and versioned protocol support.
   - The README makes clear that the public API is OpenAI-compatible, so external clients can talk to `localhost:9337/v1` without mesh-specific knowledge.
   - That compatibility layer is also what enables launcher integrations and agent tools to use the mesh as a drop-in backend.

10. **Runtime and system management**
   - `mesh-llm/src/system` covers hardware detection, benchmarking, and auto-update behavior.
   - `mesh-llm/src/runtime` includes local runtime control and proxy-facing logic for starting/stopping models on a node.
   - The architecture treats local node lifecycle as separate from mesh-wide coordination; stage-one load/unload is intentionally node-local.

11. **Frontend console**
   - `mesh-llm/ui` is a separate Vite/TypeScript app with its own package lock and Tailwind-based component set.
   - It is not the control plane; it is a visualization and interaction layer for status, model selection, and chat.
   - The console is served alongside the Rust backend and depends on backend APIs rather than maintaining independent state.

12. **Deployment and distribution**
   - `install.sh`, `Justfile`, `scripts/`, and `.github/workflows/` form the build/release pipeline.
   - `dist/` contains service definitions for macOS and Linux background installs.
   - `fly/` and `relay/` indicate deployment support for hosted/public mesh infrastructure, likely used for discovery or transport relaying.

13. **Testing, evaluation, and benchmarking**
   - `tests/` plus shell-based test scripts verify blackboard and compatibility behavior.
   - `evals/` contains scenario-driven evaluation harnesses for agent workflows and latency benchmarks.
   - `benchmarks/` includes native performance probes for different GPU backends, reflecting the repo’s emphasis on hardware-sensitive behavior.

14. **Important architectural boundary to capture in a diagram**
   - The main split is between:
     - **Control/UI/CLI**: user commands, status, model management, web console.
     - **Mesh/runtime**: discovery, routing, peer selection, tunnels, load balancing.
     - **Inference backends**: local llama.cpp-based servers, RPC workers, and distributed splits.
   - The system is designed so a client talks to one local API, while the backend transparently decides whether the model runs locally, is pipelined across peers, or is sharded by experts.

Main entry points: [`mesh-llm/Cargo.toml`](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/Cargo.toml) (mesh-llm), [`mesh-llm/src/api/mod.rs`](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/api/mod.rs) (HTTP API), [`mesh-llm/src/network/mod.rs`](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/network/mod.rs) (Mesh network), [`mesh-llm/src/runtime/mod.rs`](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/runtime/mod.rs) (Runtime)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core runtime"]
  node_main_crate["mesh-llm<br/>Rust runtime<br/>[Cargo.toml]"]
  node_cli["CLI<br/>operator commands<br/>[mod.rs]"]
  node_api["HTTP API<br/>local control API<br/>[mod.rs]"]
  node_inference["Inference<br/>launch orchestration<br/>[mod.rs]"]
  node_runtime["Runtime<br/>node lifecycle<br/>[mod.rs]"]
  node_network["Mesh network<br/>routing layer<br/>[mod.rs]"]
  node_models["Models<br/>catalog and resolution<br/>[mod.rs]"]
  node_protocol["Protocol<br/>compatibility layer<br/>[mod.rs]"]
  node_system["System<br/>hardware and update<br/>[mod.rs]"]
  node_mesh_mod["Mesh core<br/>coordination boundary<br/>[mod.rs]"]
  node_proto_defs["Protos<br/>rpc contracts<br/>[node.proto]"]
end

subgraph group_extension["Extensions"]
  node_plugins["Plugins<br/>plugin host<br/>[mod.rs]"]
  node_plugin_runtime["Plugin runtime<br/>plugin bridge<br/>[mod.rs]"]
  node_blackboard["Blackboard<br/>shared feed<br/>[mod.rs]"]
end

subgraph group_ui["User interface"]
  node_ui_app["Web console<br/>TypeScript UI<br/>[App.tsx]"]
end

subgraph group_ops["Build and deploy"]
  node_deploy["Release ops<br/>build and packaging<br/>[build-release.sh]"]
  node_evals["Evals<br/>workflow harness<br/>[README.md]"]
  node_benchmarks["Benchmarks<br/>native probes"]
  node_relay["Relay<br/>hosted infra<br/>[README.md]"]
end

node_main_crate -->|"exposes"| node_cli
node_main_crate -->|"serves"| node_api
node_main_crate -->|"orchestrates"| node_inference
node_main_crate -->|"manages"| node_runtime
node_main_crate -->|"routes"| node_network
node_main_crate -->|"resolves"| node_models
node_main_crate -->|"translates"| node_protocol
node_main_crate -->|"detects"| node_system
node_main_crate -->|"coordinates"| node_mesh_mod
node_api -->|"consumed by"| node_ui_app
node_api -->|"compatible with"| node_protocol
node_cli -->|"controls"| node_runtime
node_inference -->|"selects"| node_models
node_inference -->|"launches via"| node_runtime
node_runtime -->|"uses"| node_network
node_network -->|"implements"| node_mesh_mod
node_plugins -->|"loads"| node_plugin_runtime
node_plugins -->|"includes"| node_blackboard
node_blackboard -->|"exposed by"| node_api
node_plugin_runtime -->|"contracts"| node_proto_defs
node_network -->|"contracts"| node_proto_defs
node_deploy -->|"packages"| node_main_crate
node_deploy -->|"bundles"| node_ui_app
node_evals -->|"verifies"| node_api
node_benchmarks -->|"measures"| node_system
node_relay -->|"supports"| node_network

click node_main_crate "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/Cargo.toml"
click node_cli "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/cli/mod.rs"
click node_api "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/api/mod.rs"
click node_inference "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/inference/mod.rs"
click node_runtime "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/runtime/mod.rs"
click node_network "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/network/mod.rs"
click node_models "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/models/mod.rs"
click node_protocol "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/protocol/mod.rs"
click node_system "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/system/mod.rs"
click node_mesh_mod "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/mesh/mod.rs"
click node_plugins "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/plugins/mod.rs"
click node_plugin_runtime "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/plugin/mod.rs"
click node_blackboard "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/plugins/blackboard/mod.rs"
click node_ui_app "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/ui/src/App.tsx"
click node_deploy "https://github.com/michaelneale/mesh-llm/blob/main/scripts/build-release.sh"
click node_evals "https://github.com/michaelneale/mesh-llm/blob/main/evals/README.md"
click node_benchmarks "https://github.com/michaelneale/mesh-llm/blob/main/benchmarks/membench-fingerprint.cu"
click node_relay "https://github.com/michaelneale/mesh-llm/blob/main/relay/README.md"
click node_proto_defs "https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/proto/node.proto"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_crate,node_cli,node_api,node_inference,node_runtime,node_network,node_models,node_protocol,node_system,node_mesh_mod,node_proto_defs toneBlue
class node_plugins,node_plugin_runtime,node_blackboard toneAmber
class node_ui_app toneMint
class node_deploy,node_evals,node_benchmarks,node_relay toneRose
```

## Components

### Core runtime

Rust crate that serves the API, CLI, mesh, and inference orchestration.

- [mesh-llm](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/Cargo.toml): Rust runtime · Primary executable crate coordinating local serving and mesh-aware operation. · `mesh-llm/Cargo.toml`
- [CLI](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/cli/mod.rs): operator commands · Install, run, join, download, status, and model-management entry points. · `mesh-llm/src/cli/mod.rs`
- [HTTP API](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/api/mod.rs): local control API · OpenAI-compatible serving surface plus local management and status routes. · `mesh-llm/src/api/mod.rs`
- [Inference](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/inference/mod.rs): launch orchestration · Coordinates model startup and execution strategy selection. · `mesh-llm/src/inference/mod.rs`
- [Runtime](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/runtime/mod.rs): node lifecycle · Local node control, discovery, and proxy-facing runtime state. · `mesh-llm/src/runtime/mod.rs`
- [Mesh network](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/network/mod.rs): routing layer · Peer selection, routing, proxying, tunnels, and message rewriting. · `mesh-llm/src/network/mod.rs`
- [Models](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/models/mod.rs): catalog and resolution · Catalog, inventory, GGUF handling, topology, and model decisions. · `mesh-llm/src/models/mod.rs`
- [Protocol](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/protocol/mod.rs): compatibility layer · Versioned protocol conversion and public API compatibility support. · `mesh-llm/src/protocol/mod.rs`
- [System](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/system/mod.rs): hardware and update · Hardware detection, benchmarking, and autoupdate support. · `mesh-llm/src/system/mod.rs`
- [Mesh core](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/mesh/mod.rs): coordination boundary · Top-level mesh coordination and cross-node behavior boundary. · `mesh-llm/src/mesh/mod.rs`
- [Protos](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/proto/node.proto): rpc contracts · Cross-process contracts used by mesh and plugin layers. · `mesh-llm/proto/node.proto`

### Extensions

Plugin and blackboard integration boundaries.

- [Plugins](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/plugins/mod.rs): plugin host · Plugin subsystem and built-in plugin integrations. · `mesh-llm/src/plugins/mod.rs`
- [Plugin runtime](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/plugin/mod.rs): plugin bridge · Transport, MCP, config, and support for plugin execution. · `mesh-llm/src/plugin/mod.rs`
- [Blackboard](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/src/plugins/blackboard/mod.rs): shared feed · Ephemeral shared message feed exposed as mesh feature and MCP server. · `mesh-llm/src/plugins/blackboard/mod.rs`

### User interface

TypeScript console for monitoring and interaction.

- [Web console](https://github.com/michaelneale/mesh-llm/blob/main/mesh-llm/ui/src/App.tsx): TypeScript UI · Vite-based frontend for topology, status, and chat interaction. · `mesh-llm/ui/src/App.tsx`

### Build and deploy

Packaging, release, relay, eval, and benchmark assets.

- [Release ops](https://github.com/michaelneale/mesh-llm/blob/main/scripts/build-release.sh): build and packaging · Scripts, CI, install, and packaging paths for shipping the system. · `scripts/build-release.sh`
- [Evals](https://github.com/michaelneale/mesh-llm/blob/main/evals/README.md): workflow harness · Scenario-driven evaluation, latency measurement, and compatibility smoke tests. · `evals/README.md`
- [Benchmarks](https://github.com/michaelneale/mesh-llm/blob/main/benchmarks/membench-fingerprint.cu): native probes · GPU and memory benchmark sources used to validate hardware-sensitive behavior. · `benchmarks/membench-fingerprint.cu`
- [Relay](https://github.com/michaelneale/mesh-llm/blob/main/relay/README.md): hosted infra · Relay deployment support for public mesh connectivity. · `relay/README.md`

## Connections

- mesh-llm → CLI: exposes · The main binary wires command-line entry points into the runtime.
- mesh-llm → HTTP API: serves · The crate hosts the HTTP control and OpenAI-compatible API.
- mesh-llm → Inference: orchestrates · The executable coordinates model launch and execution planning.
- mesh-llm → Runtime: manages · Node-local lifecycle and proxy control live in runtime code.
- mesh-llm → Mesh network: routes · Distributed request routing and peer communication are handled here.
- mesh-llm → Models: resolves · Model lookup and local inventory decisions feed serving choices.
- mesh-llm → Protocol: translates · Compatibility and versioned protocol conversion sit on the serving boundary.
- mesh-llm → System: detects · Hardware and update logic inform runtime decisions.
- mesh-llm → Mesh core: coordinates · Mesh-specific behavior is separated from local control concerns.
- HTTP API → Web console: consumed by · The console uses backend status, events, and chat endpoints.
- HTTP API → Protocol: compatible with · The HTTP surface supports OpenAI-style clients.
- CLI → Runtime: controls · Operator commands start, stop, and inspect the local node.
- Inference → Models: selects · Serving strategy depends on model fit, topology, and capabilities.
- Inference → Runtime: launches via · Model execution is started and supervised through node runtime.
- Runtime → Mesh network: uses · Local runtime delegates peer discovery and proxy behavior to the mesh layer.
- Mesh network → Mesh core: implements · Peer routing and request rewriting are the mesh boundary.
- Plugins → Plugin runtime: loads · Built-in and external plugins run through the plugin bridge.
- Plugins → Blackboard: includes · The blackboard feature is a first-class plugin-backed integration.
- Blackboard → HTTP API: exposed by · Blackboard functionality is available through API routes and MCP.
- Plugin runtime → Protos: contracts · Plugin execution relies on shared protobuf interfaces.
- Mesh network → Protos: contracts · Cross-process mesh communication is defined by protobuf contracts.
- Release ops → mesh-llm: packages · Build and release scripts assemble the Rust service.
- Release ops → Web console: bundles · Release tooling also builds the frontend console.
- Evals → HTTP API: verifies · Evals and smoke tests exercise compatibility and agent workflows.
- Benchmarks → System: measures · Native probes validate hardware and backend performance assumptions.
- Relay → Mesh network: supports · Hosted relay infrastructure extends mesh connectivity.
