# rizalxplo-dotcom/freetoken architecture

> Architecture diagram of the GitHub repository rizalxplo-dotcom/freetoken, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rizalxplo-dotcom/freetoken
- Repository: https://github.com/rizalxplo-dotcom/freetoken

## Overview

FreeToken is an edge-native MoE inference and serving engine for running large open-weight language models on consumer hardware. Its main workflow accepts API requests, tokenizes and schedules them, executes model inference using GPU/CPU MoE resources and attention caches, then streams generated output back through compatible APIs. The repository also provides an offline LLM interface, a persistent daemon control plane, and checkpoint conversion. The README documents adaptive CPU–GPU execution, semantic-aware caching, and elastic memory management; sampled excerpts do not establish every internal connection, so uncertain links are omitted.

Main entry points: [`python/freetoken/server/api_server.py`](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/server/api_server.py) (HTTP API), [`python/freetoken/scheduler/scheduler.py`](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/scheduler/scheduler.py) (Request Scheduler), [`python/freetoken/engine/engine.py`](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/engine/engine.py) (Inference Engine), [`python/freetoken/models`](https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/models) (Model Architectures)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Request Serving"]
  node_api_server["HTTP API<br/>[api_server.py]"]
  node_generation["Generation<br/>[generation.py]"]
  node_tokenizer["Tokenizer Service<br/>[server.py]"]
  node_scheduler["Request Scheduler<br/>[scheduler.py]"]
end

subgraph group_runtime["Inference Runtime"]
  node_engine["Inference Engine<br/>[engine.py]"]
  node_model_loader["Model Loader<br/>[loader.py]"]
  node_model_arch["Model Architectures"]
  node_layers["Model Layers"]
  node_attention["Attention Backends"]
  node_kernels["GPU Kernels"]
  node_moe["MoE Execution"]
end

subgraph group_resources["Memory Resources"]
  node_expert_banks[("Expert Banks<br/>[expert_banks.py]")]
  node_kv_cache[("KV Caches")]
  node_cache_scheduler["Cache Scheduling<br/>[cache.py]"]
end

subgraph group_operations["Operations"]
  node_checkpoint["Checkpoint Conversion<br/>[convert.py]"]
  node_daemon["Daemon Control<br/>[server.py]"]
  node_serve_manager["Serve Lifecycle<br/>[serve_manager.py]"]
  node_control_api["Control API<br/>[app.py]"]
end

subgraph group_interfaces["User Interfaces"]
  node_shell["Interactive Shell<br/>[client.py]"]
  node_offline_llm["Offline LLM API<br/>[llm.py]"]
end

node_api_client(("API Client"))
node_api_result(("Generated Response"))

node_api_client -->|"sends requests"| node_api_server
node_api_server -->|"dispatches"| node_generation
node_generation -->|"sends messages"| node_tokenizer
node_tokenizer -->|"forwards tokens"| node_scheduler
node_scheduler -->|"schedules inference"| node_engine
node_engine -->|"loads model"| node_model_loader
node_engine -->|"loads banks"| node_expert_banks
node_model_loader -->|"selects architecture"| node_model_arch
node_model_arch -->|"uses layers"| node_layers
node_model_arch -->|"uses backend"| node_attention
node_attention -->|"stores KV"| node_kv_cache
node_layers -->|"runs experts"| node_moe
node_moe -->|"uses banks"| node_expert_banks
node_moe -->|"dispatches kernels"| node_kernels
node_attention -->|"dispatches kernels"| node_kernels
node_scheduler -->|"manages cache"| node_cache_scheduler
node_cache_scheduler -->|"allocates cache"| node_kv_cache
node_api_server -->|"streams response"| node_api_result
node_api_server -.->|"offers conversion"| node_checkpoint
node_daemon -->|"serves control"| node_control_api
node_control_api -->|"controls lifecycle"| node_serve_manager
node_serve_manager -->|"launches serve"| node_api_server
node_shell -->|"calls API"| node_api_server
node_offline_llm -->|"runs requests"| node_scheduler

click node_api_server "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/server/api_server.py"
click node_generation "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/server/generation.py"
click node_tokenizer "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/tokenizer/server.py"
click node_scheduler "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/scheduler/scheduler.py"
click node_engine "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/engine/engine.py"
click node_model_loader "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/models/loader.py"
click node_model_arch "https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/models"
click node_layers "https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/layers"
click node_attention "https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/attention"
click node_kernels "https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/kernel/triton"
click node_moe "https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/moe"
click node_expert_banks "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/moe/expert_banks.py"
click node_kv_cache "https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/kvcache"
click node_cache_scheduler "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/scheduler/cache.py"
click node_checkpoint "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/checkpoint/convert.py"
click node_daemon "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/daemon/server.py"
click node_serve_manager "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/daemon/serve_manager.py"
click node_control_api "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/daemon/app.py"
click node_shell "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/shell/client.py"
click node_offline_llm "https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/llm/llm.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api_server,node_generation,node_tokenizer,node_scheduler,node_api_client toneBlue
class node_engine,node_model_loader,node_model_arch,node_layers,node_attention,node_kernels,node_moe toneAmber
class node_expert_banks,node_kv_cache,node_cache_scheduler toneMint
class node_checkpoint,node_daemon,node_serve_manager,node_control_api toneRose
class node_shell,node_offline_llm,node_api_result toneIndigo
```

## Components

### Request Serving

- [HTTP API](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/server/api_server.py): `python/freetoken/server/api_server.py`
- [Generation](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/server/generation.py): `python/freetoken/server/generation.py`
- [Tokenizer Service](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/tokenizer/server.py): `python/freetoken/tokenizer/server.py`
- [Request Scheduler](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/scheduler/scheduler.py): `python/freetoken/scheduler/scheduler.py`

### Inference Runtime

- [Inference Engine](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/engine/engine.py): `python/freetoken/engine/engine.py`
- [Model Loader](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/models/loader.py): `python/freetoken/models/loader.py`
- [Model Architectures](https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/models): `python/freetoken/models`
- [Model Layers](https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/layers): `python/freetoken/layers`
- [Attention Backends](https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/attention): `python/freetoken/attention`
- [GPU Kernels](https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/kernel/triton): `python/freetoken/kernel/triton`
- [MoE Execution](https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/moe): `python/freetoken/moe`

### Memory Resources

- [Expert Banks](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/moe/expert_banks.py): `python/freetoken/moe/expert_banks.py`
- [KV Caches](https://github.com/rizalxplo-dotcom/freetoken/tree/main/python/freetoken/kvcache): `python/freetoken/kvcache`
- [Cache Scheduling](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/scheduler/cache.py): `python/freetoken/scheduler/cache.py`

### Operations

- [Checkpoint Conversion](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/checkpoint/convert.py): `python/freetoken/checkpoint/convert.py`
- [Daemon Control](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/daemon/server.py): `python/freetoken/daemon/server.py`
- [Serve Lifecycle](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/daemon/serve_manager.py): `python/freetoken/daemon/serve_manager.py`
- [Control API](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/daemon/app.py): `python/freetoken/daemon/app.py`

### User Interfaces

- [Interactive Shell](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/shell/client.py): `python/freetoken/shell/client.py`
- [Offline LLM API](https://github.com/rizalxplo-dotcom/freetoken/blob/main/python/freetoken/llm/llm.py): `python/freetoken/llm/llm.py`

### Other components

- **API Client**
- **Generated Response**

## Connections

- API Client → HTTP API: sends requests
- HTTP API → Generation: dispatches
- Generation → Tokenizer Service: sends messages
- Tokenizer Service → Request Scheduler: forwards tokens
- Request Scheduler → Inference Engine: schedules inference
- Inference Engine → Model Loader: loads model
- Inference Engine → Expert Banks: loads banks
- Model Loader → Model Architectures: selects architecture
- Model Architectures → Model Layers: uses layers
- Model Architectures → Attention Backends: uses backend
- Attention Backends → KV Caches: stores KV
- Model Layers → MoE Execution: runs experts
- MoE Execution → Expert Banks: uses banks
- MoE Execution → GPU Kernels: dispatches kernels
- Attention Backends → GPU Kernels: dispatches kernels
- Request Scheduler → Cache Scheduling: manages cache
- Cache Scheduling → KV Caches: allocates cache
- HTTP API → Generated Response: streams response
- HTTP API → Checkpoint Conversion: offers conversion
- Daemon Control → Control API: serves control
- Control API → Serve Lifecycle: controls lifecycle
- Serve Lifecycle → HTTP API: launches serve
- Interactive Shell → HTTP API: calls API
- Offline LLM API → Request Scheduler: runs requests