# edge0-ai/edge0 architecture

> Architecture diagram of the GitHub repository edge0-ai/edge0, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/edge0-ai/edge0
- Repository: https://github.com/edge0-ai/edge0

## Overview

edge0 is a streaming mixture-of-experts (MoE) inference framework. A user selects a checkpoint through the CLI or Python API; tier resolution and engine setup lead to model loading, after which chat requests generate tokens through the model’s inference path. The main product mechanisms are SSD-backed expert streaming, prerouter prediction to stage experts early, and unmerged Recover-LoRA adapters. The README documents these capabilities; sampled source confirms parts of the server, adapter, routing, and staging behavior, but not every engine-to-component call.

Main entry points: [`src/edge0/streaming/layer.py`](https://github.com/edge0-ai/edge0/blob/main/src/edge0/streaming/layer.py) (Streaming experts), [`src/edge0/engine/qwen.py`](https://github.com/edge0-ai/edge0/blob/main/src/edge0/engine/qwen.py) (Qwen engine), [`src/edge0/engine/ling.py`](https://github.com/edge0-ai/edge0/blob/main/src/edge0/engine/ling.py) (Ling engine), [`src/edge0/server/chat.py`](https://github.com/edge0-ai/edge0/blob/main/src/edge0/server/chat.py) (Chat session)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User interfaces"]
  node_cli["CLI<br/>[cli.py]"]
  node_server_app["HTTP server<br/>[app.py]"]
  node_chat["Chat session<br/>[chat.py]"]
end

subgraph group_runtime["Inference runtime"]
  node_registry["Tier registry<br/>[registry.py]"]
  node_config["Model configuration<br/>[base.py]"]
  node_engine_base["Engine base<br/>[base.py]"]
  node_engine_qwen["Qwen engine<br/>[qwen.py]"]
  node_engine_ling["Ling engine<br/>[ling.py]"]
  node_sampling["Token sampling<br/>[sampling.py]"]
end

subgraph group_model["Models and adapters"]
  node_model_35b["35B model tier<br/>[__init__.py]"]
  node_model_8b["8B model tier<br/>[__init__.py]"]
  node_lora["Recover-LoRA<br/>[lora.py]"]
  node_prerouter["Prerouter staging<br/>[stager.py]"]
  node_routing["MoE routing<br/>[routing.py]"]
end

subgraph group_storage["Streaming and storage"]
  node_stream_layer["Streaming experts<br/>[layer.py]"]
  node_stream_cache["Expert cache<br/>[cache.py]"]
  node_stream_mmap["Memory-mapped weights<br/>[mmap.py]"]
end

subgraph group_backend["Backend implementation"]
  node_mlx_backend["MLX backend<br/>[backend.py]"]
  node_mlx_io["Checkpoint loading<br/>[io.py]"]
  node_mlx_impl["Model implementations"]
end

node_user(("User"))
node_http_client(("HTTP client"))
node_checkpoint["Model checkpoint"]

node_user -->|"runs command"| node_cli
node_user -->|"submits prompt"| node_chat
node_http_client -->|"sends request"| node_server_app
node_cli -->|"resolves tier"| node_registry
node_cli -->|"starts session"| node_chat
node_cli -->|"loads engine"| node_engine_base
node_registry -->|"selects config"| node_config
node_config -->|"uses tier defaults"| node_model_35b
node_config -->|"uses tier defaults"| node_model_8b
node_engine_base -->|"dispatches family"| node_engine_qwen
node_engine_base -->|"dispatches family"| node_engine_ling
node_engine_qwen -->|"uses backend"| node_mlx_backend
node_engine_ling -->|"uses backend"| node_mlx_backend
node_engine_qwen -->|"installs streaming"| node_stream_layer
node_engine_ling -->|"installs streaming"| node_stream_layer
node_engine_qwen -->|"installs adapters"| node_lora
node_engine_ling -->|"installs adapters"| node_lora
node_engine_qwen -->|"stages predictions"| node_prerouter
node_engine_ling -->|"stages predictions"| node_prerouter
node_prerouter -->|"stages experts"| node_stream_layer
node_stream_layer -->|"selects experts"| node_routing
node_stream_layer -->|"uses cache"| node_stream_cache
node_stream_layer -->|"reads weights"| node_stream_mmap
node_mlx_backend -->|"loads tensors"| node_mlx_io
node_mlx_backend -->|"provides kernels"| node_mlx_impl
node_mlx_io -->|"reads files"| node_checkpoint
node_chat -->|"generates tokens"| node_engine_base
node_chat -->|"samples output"| node_sampling
node_server_app -->|"dispatches chat"| node_chat
node_stream_mmap -->|"maps weights"| node_checkpoint

click node_cli "https://github.com/edge0-ai/edge0/blob/main/src/edge0/cli.py"
click node_server_app "https://github.com/edge0-ai/edge0/blob/main/src/edge0/server/app.py"
click node_chat "https://github.com/edge0-ai/edge0/blob/main/src/edge0/server/chat.py"
click node_registry "https://github.com/edge0-ai/edge0/blob/main/src/edge0/registry.py"
click node_config "https://github.com/edge0-ai/edge0/blob/main/src/edge0/models/base.py"
click node_engine_base "https://github.com/edge0-ai/edge0/blob/main/src/edge0/engine/base.py"
click node_engine_qwen "https://github.com/edge0-ai/edge0/blob/main/src/edge0/engine/qwen.py"
click node_engine_ling "https://github.com/edge0-ai/edge0/blob/main/src/edge0/engine/ling.py"
click node_sampling "https://github.com/edge0-ai/edge0/blob/main/src/edge0/sampling.py"
click node_model_35b "https://github.com/edge0-ai/edge0/blob/main/src/edge0/models/edge0_35b/__init__.py"
click node_model_8b "https://github.com/edge0-ai/edge0/blob/main/src/edge0/models/edge0_8b/__init__.py"
click node_lora "https://github.com/edge0-ai/edge0/blob/main/src/edge0/adapters/lora.py"
click node_prerouter "https://github.com/edge0-ai/edge0/blob/main/src/edge0/prerouter/stager.py"
click node_routing "https://github.com/edge0-ai/edge0/blob/main/src/edge0/moe/routing.py"
click node_stream_layer "https://github.com/edge0-ai/edge0/blob/main/src/edge0/streaming/layer.py"
click node_stream_cache "https://github.com/edge0-ai/edge0/blob/main/src/edge0/streaming/cache.py"
click node_stream_mmap "https://github.com/edge0-ai/edge0/blob/main/src/edge0/streaming/mmap.py"
click node_mlx_backend "https://github.com/edge0-ai/edge0/blob/main/src/edge0/backends/mlx/backend.py"
click node_mlx_io "https://github.com/edge0-ai/edge0/blob/main/src/edge0/backends/mlx/io.py"
click node_mlx_impl "https://github.com/edge0-ai/edge0/tree/main/src/edge0/backends/mlx/_impl"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_server_app,node_chat,node_user,node_http_client toneBlue
class node_registry,node_config,node_engine_base,node_engine_qwen,node_engine_ling,node_sampling toneAmber
class node_model_35b,node_model_8b,node_lora,node_prerouter,node_routing toneMint
class node_stream_layer,node_stream_cache,node_stream_mmap toneRose
class node_mlx_backend,node_mlx_io,node_mlx_impl,node_checkpoint toneIndigo
```

## Components

### User interfaces

- [CLI](https://github.com/edge0-ai/edge0/blob/main/src/edge0/cli.py): `src/edge0/cli.py`
- [HTTP server](https://github.com/edge0-ai/edge0/blob/main/src/edge0/server/app.py): `src/edge0/server/app.py`
- [Chat session](https://github.com/edge0-ai/edge0/blob/main/src/edge0/server/chat.py): `src/edge0/server/chat.py`

### Inference runtime

- [Tier registry](https://github.com/edge0-ai/edge0/blob/main/src/edge0/registry.py): `src/edge0/registry.py`
- [Model configuration](https://github.com/edge0-ai/edge0/blob/main/src/edge0/models/base.py): `src/edge0/models/base.py`
- [Engine base](https://github.com/edge0-ai/edge0/blob/main/src/edge0/engine/base.py): `src/edge0/engine/base.py`
- [Qwen engine](https://github.com/edge0-ai/edge0/blob/main/src/edge0/engine/qwen.py): `src/edge0/engine/qwen.py`
- [Ling engine](https://github.com/edge0-ai/edge0/blob/main/src/edge0/engine/ling.py): `src/edge0/engine/ling.py`
- [Token sampling](https://github.com/edge0-ai/edge0/blob/main/src/edge0/sampling.py): `src/edge0/sampling.py`

### Models and adapters

- [35B model tier](https://github.com/edge0-ai/edge0/blob/main/src/edge0/models/edge0_35b/__init__.py): `src/edge0/models/edge0_35b/__init__.py`
- [8B model tier](https://github.com/edge0-ai/edge0/blob/main/src/edge0/models/edge0_8b/__init__.py): `src/edge0/models/edge0_8b/__init__.py`
- [Recover-LoRA](https://github.com/edge0-ai/edge0/blob/main/src/edge0/adapters/lora.py): `src/edge0/adapters/lora.py`
- [Prerouter staging](https://github.com/edge0-ai/edge0/blob/main/src/edge0/prerouter/stager.py): `src/edge0/prerouter/stager.py`
- [MoE routing](https://github.com/edge0-ai/edge0/blob/main/src/edge0/moe/routing.py): `src/edge0/moe/routing.py`

### Streaming and storage

- [Streaming experts](https://github.com/edge0-ai/edge0/blob/main/src/edge0/streaming/layer.py): `src/edge0/streaming/layer.py`
- [Expert cache](https://github.com/edge0-ai/edge0/blob/main/src/edge0/streaming/cache.py): `src/edge0/streaming/cache.py`
- [Memory-mapped weights](https://github.com/edge0-ai/edge0/blob/main/src/edge0/streaming/mmap.py): `src/edge0/streaming/mmap.py`

### Backend implementation

- [MLX backend](https://github.com/edge0-ai/edge0/blob/main/src/edge0/backends/mlx/backend.py): `src/edge0/backends/mlx/backend.py`
- [Checkpoint loading](https://github.com/edge0-ai/edge0/blob/main/src/edge0/backends/mlx/io.py): `src/edge0/backends/mlx/io.py`
- [Model implementations](https://github.com/edge0-ai/edge0/tree/main/src/edge0/backends/mlx/_impl): `src/edge0/backends/mlx/_impl`

### Other components

- **User**
- **HTTP client**
- **Model checkpoint**

## Connections

- User → CLI: runs command
- User → Chat session: submits prompt
- HTTP client → HTTP server: sends request
- CLI → Tier registry: resolves tier
- CLI → Chat session: starts session
- CLI → Engine base: loads engine
- Tier registry → Model configuration: selects config
- Model configuration → 35B model tier: uses tier defaults
- Model configuration → 8B model tier: uses tier defaults
- Engine base → Qwen engine: dispatches family
- Engine base → Ling engine: dispatches family
- Qwen engine → MLX backend: uses backend
- Ling engine → MLX backend: uses backend
- Qwen engine → Streaming experts: installs streaming
- Ling engine → Streaming experts: installs streaming
- Qwen engine → Recover-LoRA: installs adapters
- Ling engine → Recover-LoRA: installs adapters
- Qwen engine → Prerouter staging: stages predictions
- Ling engine → Prerouter staging: stages predictions
- Prerouter staging → Streaming experts: stages experts
- Streaming experts → MoE routing: selects experts
- Streaming experts → Expert cache: uses cache
- Streaming experts → Memory-mapped weights: reads weights
- MLX backend → Checkpoint loading: loads tensors
- MLX backend → Model implementations: provides kernels
- Checkpoint loading → Model checkpoint: reads files
- Chat session → Engine base: generates tokens
- Chat session → Token sampling: samples output
- HTTP server → Chat session: dispatches chat
- Memory-mapped weights → Model checkpoint: maps weights