How rizalxplo-dotcom/freetoken works
FreeToken is an edge-native MoE inference and serving engine for running large open-weight language models on consumer hardware. Its main workflow accepts API requests, tokenizes and schedules them, executes model inference using GPU/CPU MoE resources and attention caches, then streams generated output back through compatible APIs. The repository also provides an offline LLM interface, a persistent daemon control plane, and checkpoint conversion. The README documents adaptive CPU–GPU execution, semantic-aware caching, and elastic memory management; sampled excerpts do not establish every internal connection, so uncertain links are omitted.
Main entry points: api_server.py HTTP API · scheduler.py Request Scheduler · engine.py Inference Engine · models Model Architectures
Components 22
Request Serving
- HTTP API
python/freetoken/server/api_server.py - Generation
python/freetoken/server/generation.py - Tokenizer Service
python/freetoken/tokenizer/server.py - Request Scheduler
python/freetoken/scheduler/scheduler.py
Inference Runtime
- Inference Engine
python/freetoken/engine/engine.py - Model Loader
python/freetoken/models/loader.py - Model Architectures
python/freetoken/models - Model Layers
python/freetoken/layers - Attention Backends
python/freetoken/attention - GPU Kernels
python/freetoken/kernel/triton - MoE Execution
python/freetoken/moe
Memory Resources
- Expert Banks
python/freetoken/moe/expert_banks.py - KV Caches
python/freetoken/kvcache - Cache Scheduling
python/freetoken/scheduler/cache.py
Operations
- Checkpoint Conversion
python/freetoken/checkpoint/convert.py - Daemon Control
python/freetoken/daemon/server.py - Serve Lifecycle
python/freetoken/daemon/serve_manager.py - Control API
python/freetoken/daemon/app.py
User Interfaces
- Interactive Shell
python/freetoken/shell/client.py - Offline LLM API
python/freetoken/llm/llm.py
Other components
- API Client
- Generated Response
Connections 24
- API Client → HTTP API: sends requests
- HTTP API → Generation: dispatches
- Generation → Tokenizer Service: sends messages
- Tokenizer Service → Request Scheduler: forwards tokens
- Request Scheduler → Inference Engine: schedules inference
- Inference Engine → Model Loader: loads model
- Inference Engine → Expert Banks: loads banks
- Model Loader → Model Architectures: selects architecture
- Model Architectures → Model Layers: uses layers
- Model Architectures → Attention Backends: uses backend
- Attention Backends → KV Caches: stores KV
- Model Layers → MoE Execution: runs experts
- MoE Execution → Expert Banks: uses banks
- MoE Execution → GPU Kernels: dispatches kernels
- Attention Backends → GPU Kernels: dispatches kernels
- Request Scheduler → Cache Scheduling: manages cache
- Cache Scheduling → KV Caches: allocates cache
- HTTP API → Generated Response: streams response
- HTTP API → Checkpoint Conversion: offers conversion
- Daemon Control → Control API: serves control
- Control API → Serve Lifecycle: controls lifecycle
- Serve Lifecycle → HTTP API: launches serve
- Interactive Shell → HTTP API: calls API
- Offline LLM API → Request Scheduler: runs requests