# ammar-abagumbul/tokencake architecture

> Architecture diagram of the GitHub repository ammar-abagumbul/tokencake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ammar-abagumbul/tokencake
- Repository: https://github.com/ammar-abagumbul/tokencake

## Overview

TokenCake is an agent-aware KV-cache scheduler for multi-agent LLM serving. In the documented current milestone, an agent workflow follows a dependency DAG, calls an LLM and external tools, and brackets tool stalls with frontend events. The frontend delegates those events to the temporal controller, which forecasts stall duration, gates offload decisions, and plans cache uploads; the connector integrates KV migration with vLLM. The README also describes a spatial scheduler, but the stated milestone targets temporal scheduling and no spatial implementation path appears in the tree. Unsampled policy wiring is shown only where documented.

Main entry points: [`agents/workflow.py`](https://github.com/ammar-abagumbul/tokencake/blob/main/agents/workflow.py) (Workflow driver), [`src/tokencake/temporal/controller.py`](https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/temporal/controller.py) (Temporal controller), [`src/tokencake/frontend/api.py`](https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/frontend/api.py) (Event API), [`src/tokencake/connector.py`](https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/connector.py) (vLLM KV connector)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workload["Agent workload"]
  node_app_catalog["Agent applications<br/>[apps.py]"]
  node_dag["Agent DAG<br/>[graph.py]"]
  node_workflow["Workflow driver<br/>[workflow.py]"]
  node_tool_exec["Tool execution<br/>[tools.py]"]
end

subgraph group_control["Frontend control"]
  node_frontend_api["Event API<br/>[api.py]"]
end

subgraph group_temporal["Temporal policy"]
  node_controller["Temporal controller<br/>[controller.py]"]
  node_forecaster["Duration forecast<br/>[forecast.py]"]
  node_offload_gate["Offload gate<br/>[gate.py]"]
  node_upload_planner["Upload planner<br/>[prefetch.py]"]
end

subgraph group_serving["KV serving"]
  node_offload_manager[("CPU block manager<br/>[offload_manager.py]")]
  node_connector["vLLM KV connector<br/>[connector.py]"]
end

node_app_user(("Application caller"))
node_llm["vLLM server"]

node_app_user -->|"starts workflow"| node_workflow
node_app_catalog -->|"builds DAG"| node_dag
node_workflow -->|"executes DAG"| node_dag
node_workflow -->|"requests inference"| node_llm
node_workflow -->|"executes tools"| node_tool_exec
node_workflow -->|"posts events"| node_frontend_api
node_frontend_api -->|"dispatches events"| node_controller
node_controller -->|"predicts duration"| node_forecaster
node_controller -->|"requests decision"| node_offload_gate
node_controller -->|"manages cache"| node_offload_manager
node_controller -->|"plans uploads"| node_upload_planner
node_llm -.->|"invokes connector"| node_connector
node_connector -->|"tracks KV blocks"| node_offload_manager

click node_app_catalog "https://github.com/ammar-abagumbul/tokencake/blob/main/agents/apps.py"
click node_dag "https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/graph.py"
click node_workflow "https://github.com/ammar-abagumbul/tokencake/blob/main/agents/workflow.py"
click node_tool_exec "https://github.com/ammar-abagumbul/tokencake/blob/main/agents/tools.py"
click node_frontend_api "https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/frontend/api.py"
click node_controller "https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/temporal/controller.py"
click node_forecaster "https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/temporal/forecast.py"
click node_offload_gate "https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/temporal/gate.py"
click node_upload_planner "https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/temporal/prefetch.py"
click node_offload_manager "https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/offload_manager.py"
click node_connector "https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/connector.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app_catalog,node_dag,node_workflow,node_tool_exec toneBlue
class node_frontend_api toneAmber
class node_controller,node_forecaster,node_offload_gate,node_upload_planner,node_llm toneMint
class node_offload_manager,node_connector toneRose
class node_app_user toneIndigo
```

## Components

### Agent workload

- [Agent applications](https://github.com/ammar-abagumbul/tokencake/blob/main/agents/apps.py): `agents/apps.py`
- [Agent DAG](https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/graph.py): `src/tokencake/graph.py`
- [Workflow driver](https://github.com/ammar-abagumbul/tokencake/blob/main/agents/workflow.py): `agents/workflow.py`
- [Tool execution](https://github.com/ammar-abagumbul/tokencake/blob/main/agents/tools.py): `agents/tools.py`

### Frontend control

- [Event API](https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/frontend/api.py): `src/tokencake/frontend/api.py`

### Temporal policy

- [Temporal controller](https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/temporal/controller.py): `src/tokencake/temporal/controller.py`
- [Duration forecast](https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/temporal/forecast.py): `src/tokencake/temporal/forecast.py`
- [Offload gate](https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/temporal/gate.py): `src/tokencake/temporal/gate.py`
- [Upload planner](https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/temporal/prefetch.py): `src/tokencake/temporal/prefetch.py`

### KV serving

- [CPU block manager](https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/offload_manager.py): `src/tokencake/offload_manager.py`
- [vLLM KV connector](https://github.com/ammar-abagumbul/tokencake/blob/main/src/tokencake/connector.py): `src/tokencake/connector.py`

### Other components

- **Application caller**
- **vLLM server**

## Connections

- Application caller → Workflow driver: starts workflow
- Agent applications → Agent DAG: builds DAG
- Workflow driver → Agent DAG: executes DAG
- Workflow driver → vLLM server: requests inference
- Workflow driver → Tool execution: executes tools
- Workflow driver → Event API: posts events
- Event API → Temporal controller: dispatches events
- Temporal controller → Duration forecast: predicts duration
- Temporal controller → Offload gate: requests decision
- Temporal controller → CPU block manager: manages cache
- Temporal controller → Upload planner: plans uploads
- vLLM server → vLLM KV connector: invokes connector
- vLLM KV connector → CPU block manager: tracks KV blocks