How edge0-ai/edge0 works
edge0 is a streaming mixture-of-experts (MoE) inference framework. A user selects a checkpoint through the CLI or Python API; tier resolution and engine setup lead to model loading, after which chat requests generate tokens through the model’s inference path. The main product mechanisms are SSD-backed expert streaming, prerouter prediction to stage experts early, and unmerged Recover-LoRA adapters. The README documents these capabilities; sampled source confirms parts of the server, adapter, routing, and staging behavior, but not every engine-to-component call.
Main entry points: layer.py Streaming experts · qwen.py Qwen engine · ling.py Ling engine · chat.py Chat session
Components 23
User interfaces
- CLI
src/edge0/cli.py - HTTP server
src/edge0/server/app.py - Chat session
src/edge0/server/chat.py
Inference runtime
- Tier registry
src/edge0/registry.py - Model configuration
src/edge0/models/base.py - Engine base
src/edge0/engine/base.py - Qwen engine
src/edge0/engine/qwen.py - Ling engine
src/edge0/engine/ling.py - Token sampling
src/edge0/sampling.py
Models and adapters
- 35B model tier
src/edge0/models/edge0_35b/__init__.py - 8B model tier
src/edge0/models/edge0_8b/__init__.py - Recover-LoRA
src/edge0/adapters/lora.py - Prerouter staging
src/edge0/prerouter/stager.py - MoE routing
src/edge0/moe/routing.py
Streaming and storage
- Streaming experts
src/edge0/streaming/layer.py - Expert cache
src/edge0/streaming/cache.py - Memory-mapped weights
src/edge0/streaming/mmap.py
Backend implementation
- MLX backend
src/edge0/backends/mlx/backend.py - Checkpoint loading
src/edge0/backends/mlx/io.py - Model implementations
src/edge0/backends/mlx/_impl
Other components
- User
- HTTP client
- Model checkpoint
Connections 30
- User → CLI: runs command
- User → Chat session: submits prompt
- HTTP client → HTTP server: sends request
- CLI → Tier registry: resolves tier
- CLI → Chat session: starts session
- CLI → Engine base: loads engine
- Tier registry → Model configuration: selects config
- Model configuration → 35B model tier: uses tier defaults
- Model configuration → 8B model tier: uses tier defaults
- Engine base → Qwen engine: dispatches family
- Engine base → Ling engine: dispatches family
- Qwen engine → MLX backend: uses backend
- Ling engine → MLX backend: uses backend
- Qwen engine → Streaming experts: installs streaming
- Ling engine → Streaming experts: installs streaming
- Qwen engine → Recover-LoRA: installs adapters
- Ling engine → Recover-LoRA: installs adapters
- Qwen engine → Prerouter staging: stages predictions
- Ling engine → Prerouter staging: stages predictions
- Prerouter staging → Streaming experts: stages experts
- Streaming experts → MoE routing: selects experts
- Streaming experts → Expert cache: uses cache
- Streaming experts → Memory-mapped weights: reads weights
- MLX backend → Checkpoint loading: loads tensors
- MLX backend → Model implementations: provides kernels
- Checkpoint loading → Model checkpoint: reads files
- Chat session → Engine base: generates tokens
- Chat session → Token sampling: samples output
- HTTP server → Chat session: dispatches chat
- Memory-mapped weights → Model checkpoint: maps weights