# zhengjiahong-ai/llm-kv-cache-optimization architecture

> Architecture diagram of the GitHub repository zhengjiahong-ai/llm-kv-cache-optimization, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zhengjiahong-ai/llm-kv-cache-optimization
- Repository: https://github.com/zhengjiahong-ai/llm-kv-cache-optimization

## Overview

KVopt is a research scaffold for cost-aware KV-cache eviction in LLM serving, with Continuum-style retention and scheduling as supporting system mechanisms. The principal workflow is workload/request handling, cache-pressure victim selection, and block allocation or eviction; the vLLM path adapts policy decisions to native cache structures. Continuum coordinates lifecycle observations, histories, TTL-based retention, and scheduling candidates. The README documents broader capabilities than the sampled source confirms; unsampled wiring is shown only where documented, and the proposed cost-aware policy itself has no identifiable implementation in this tree.

Main entry points: [`src/kvopt/runtime/vllm/observer.py`](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/vllm/observer.py) (Runtime Hooks), [`src/kvopt/continuum/runtime.py`](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/runtime.py) (Runtime Coordinator), [`src/kvopt/runtime/simulator.py`](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/simulator.py) (Simulator), [`src/kvopt/kv_cache/manager.py`](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/kv_cache/manager.py) (Cache Manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_requests["Requests and Simulation"]
  node_workload["Workload<br/>[base.py]"]
  node_request["Request Model<br/>[request.py]"]
  node_fifo_scheduler["FIFO Scheduler<br/>[scheduler.py]"]
  node_simulator["Simulator<br/>[simulator.py]"]
end

subgraph group_eviction["Eviction and Cache"]
  node_eviction_policy["Eviction Policy<br/>[policy.py]"]
  node_cache_manager["Cache Manager<br/>[manager.py]"]
  node_cache_state["Cache State<br/>[state.py]"]
  node_dummy_policy["Dummy Policy<br/>[dummy.py]"]
end

subgraph group_continuum["Continuum Coordination"]
  node_runtime_coordinator["Runtime Coordinator<br/>[runtime.py]"]
  node_retention["Retention Manager<br/>[retention.py]"]
  node_histories[("Lifecycle Histories<br/>[history.py]")]
  node_ttl["Dynamic TTL<br/>[ttl.py]"]
  node_selection["Scheduling Policy<br/>[selection.py]"]
  node_prefill_profile["Prefill Profile<br/>[prefill_profile.py]"]
end

subgraph group_vllm["vLLM Integration"]
  node_observer["Runtime Hooks<br/>[observer.py]"]
  node_observation["Observation Extractors"]
  node_bridge["Eviction Bridge<br/>[bridge.py]"]
  node_adapter["Policy Adapter<br/>[adapter.py]"]
  node_retention_integration["Retention Adapter"]
end

subgraph group_evaluation["Metrics"]
  node_metrics[("Metrics Collector<br/>[collector.py]")]
end

node_workload_actor(("Workload Input"))
node_vllm_actor(("vLLM Runtime"))
node_native_scheduler["vLLM Scheduler"]
node_block_pool[("Native Block Pool")]
node_apc_cache[("GPU APC Cache")]

node_workload_actor -->|"supplies trace"| node_workload
node_workload -->|"yields requests"| node_request
node_simulator -->|"schedules via"| node_fifo_scheduler
node_simulator -->|"allocates blocks"| node_cache_manager
node_simulator -->|"records events"| node_metrics
node_fifo_scheduler -->|"updates status"| node_request
node_cache_manager -->|"reads state"| node_cache_state
node_cache_manager -->|"requests victims"| node_eviction_policy
node_dummy_policy -->|"implements"| node_eviction_policy
node_runtime_coordinator -->|"coordinates"| node_retention
node_runtime_coordinator -->|"records observations"| node_histories
node_runtime_coordinator -->|"derives TTL"| node_ttl
node_runtime_coordinator -->|"estimates reload"| node_prefill_profile
node_runtime_coordinator -->|"provides candidates"| node_selection
node_vllm_actor -->|"installs hooks"| node_observer
node_observer -->|"selects blocks through"| node_bridge
node_bridge -->|"adapts policy"| node_adapter
node_adapter -->|"uses policy"| node_eviction_policy
node_observer -->|"snapshots native state"| node_observation
node_observer -->|"orders waiting requests"| node_native_scheduler
node_observer -->|"hooks eviction"| node_block_pool
node_retention_integration -->|"applies retention"| node_runtime_coordinator
node_observer -->|"invokes integration"| node_retention_integration
node_block_pool -->|"manages blocks"| node_apc_cache

click node_workload "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/workload/base.py"
click node_request "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/scheduler/request.py"
click node_fifo_scheduler "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/scheduler/scheduler.py"
click node_simulator "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/simulator.py"
click node_eviction_policy "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/kv_cache/policy.py"
click node_cache_manager "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/kv_cache/manager.py"
click node_cache_state "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/kv_cache/state.py"
click node_dummy_policy "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/policies/dummy.py"
click node_runtime_coordinator "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/runtime.py"
click node_retention "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/retention.py"
click node_histories "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/history.py"
click node_ttl "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/ttl.py"
click node_selection "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/selection.py"
click node_prefill_profile "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/prefill_profile.py"
click node_observer "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/vllm/observer.py"
click node_observation "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/vllm/continuum_observation.py"
click node_bridge "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/vllm/bridge.py"
click node_adapter "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/vllm/adapter.py"
click node_retention_integration "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/vllm/retention_integration.py"
click node_metrics "https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/metrics/collector.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_workload,node_request,node_fifo_scheduler,node_simulator toneBlue
class node_eviction_policy,node_cache_manager,node_cache_state,node_dummy_policy,node_block_pool,node_apc_cache toneAmber
class node_runtime_coordinator,node_retention,node_histories,node_ttl,node_selection,node_prefill_profile toneMint
class node_observer,node_observation,node_bridge,node_adapter,node_retention_integration,node_native_scheduler toneRose
class node_metrics,node_workload_actor,node_vllm_actor toneIndigo
```

## Components

### Requests and Simulation

- [Workload](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/workload/base.py): `src/kvopt/workload/base.py`
- [Request Model](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/scheduler/request.py): `src/kvopt/scheduler/request.py`
- [FIFO Scheduler](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/scheduler/scheduler.py): `src/kvopt/scheduler/scheduler.py`
- [Simulator](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/simulator.py): `src/kvopt/runtime/simulator.py`

### Eviction and Cache

- [Eviction Policy](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/kv_cache/policy.py): `src/kvopt/kv_cache/policy.py`
- [Cache Manager](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/kv_cache/manager.py): `src/kvopt/kv_cache/manager.py`
- [Cache State](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/kv_cache/state.py): `src/kvopt/kv_cache/state.py`
- [Dummy Policy](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/policies/dummy.py): `src/kvopt/policies/dummy.py`

### Continuum Coordination

- [Runtime Coordinator](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/runtime.py): `src/kvopt/continuum/runtime.py`
- [Retention Manager](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/retention.py): `src/kvopt/continuum/retention.py`
- [Lifecycle Histories](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/history.py): `src/kvopt/continuum/history.py`
- [Dynamic TTL](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/ttl.py): `src/kvopt/continuum/ttl.py`
- [Scheduling Policy](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/selection.py): `src/kvopt/continuum/selection.py`
- [Prefill Profile](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/continuum/prefill_profile.py): `src/kvopt/continuum/prefill_profile.py`

### vLLM Integration

- [Runtime Hooks](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/vllm/observer.py): `src/kvopt/runtime/vllm/observer.py`
- [Observation Extractors](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/vllm/continuum_observation.py): `src/kvopt/runtime/vllm/continuum_observation.py`
- [Eviction Bridge](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/vllm/bridge.py): `src/kvopt/runtime/vllm/bridge.py`
- [Policy Adapter](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/vllm/adapter.py): `src/kvopt/runtime/vllm/adapter.py`
- [Retention Adapter](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/runtime/vllm/retention_integration.py): `src/kvopt/runtime/vllm/retention_integration.py`

### Metrics

- [Metrics Collector](https://github.com/zhengjiahong-ai/llm-kv-cache-optimization/blob/main/src/kvopt/metrics/collector.py): `src/kvopt/metrics/collector.py`

### Other components

- **Workload Input**
- **vLLM Runtime**
- **vLLM Scheduler**
- **Native Block Pool**
- **GPU APC Cache**

## Connections

- Workload Input → Workload: supplies trace
- Workload → Request Model: yields requests
- Simulator → FIFO Scheduler: schedules via
- Simulator → Cache Manager: allocates blocks
- Simulator → Metrics Collector: records events
- FIFO Scheduler → Request Model: updates status
- Cache Manager → Cache State: reads state
- Cache Manager → Eviction Policy: requests victims
- Dummy Policy → Eviction Policy: implements
- Runtime Coordinator → Retention Manager: coordinates
- Runtime Coordinator → Lifecycle Histories: records observations
- Runtime Coordinator → Dynamic TTL: derives TTL
- Runtime Coordinator → Prefill Profile: estimates reload
- Runtime Coordinator → Scheduling Policy: provides candidates
- vLLM Runtime → Runtime Hooks: installs hooks
- Runtime Hooks → Eviction Bridge: selects blocks through
- Eviction Bridge → Policy Adapter: adapts policy
- Policy Adapter → Eviction Policy: uses policy
- Runtime Hooks → Observation Extractors: snapshots native state
- Runtime Hooks → vLLM Scheduler: orders waiting requests
- Runtime Hooks → Native Block Pool: hooks eviction
- Retention Adapter → Runtime Coordinator: applies retention
- Runtime Hooks → Retention Adapter: invokes integration
- Native Block Pool → GPU APC Cache: manages blocks