# aiha-lab/tangram architecture

> Architecture diagram of the GitHub repository aiha-lab/tangram, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aiha-lab/tangram
- Repository: https://github.com/aiha-lab/tangram

## Overview

Tangram extends vLLM with non-uniform KV-cache compression for multi-turn LLM serving. The main path is a generation request entering the serving interface, reaching the engine, and using paged KV-cache management so compressed state can be reused across turns. README documentation describes compression scorers, budget scopes, ragged paging, memory reclamation, and compatibility with vLLM serving modes. The tree does not include the documented compression implementation paths, so those are represented as an unlocated subsystem; runtime links beyond the shown excerpts are limited to documented relationships.

Main entry points: [`vllm/v1/engine/llm_engine.py`](https://github.com/aiha-lab/tangram/blob/main/vllm/v1/engine/llm_engine.py) (V1 engine), [`vllm/v1/core/kv_cache_manager.py`](https://github.com/aiha-lab/tangram/blob/main/vllm/v1/core/kv_cache_manager.py) (Cache manager), [`vllm/entrypoints/openai/serving_engine.py`](https://github.com/aiha-lab/tangram/blob/main/vllm/entrypoints/openai/serving_engine.py) (OpenAI serving), [`vllm/entrypoints/llm.py`](https://github.com/aiha-lab/tangram/blob/main/vllm/entrypoints/llm.py) (LLM API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Serving interface"]
  node_llm_api["LLM API<br/>[llm.py]"]
  node_openai_api["OpenAI serving<br/>[serving_engine.py]"]
  node_protocol["API protocol<br/>[protocol.py]"]
end

subgraph group_engine["Request runtime"]
  node_engine["V1 engine<br/>[llm_engine.py]"]
  node_request["Request state<br/>[request.py]"]
  node_model_config["Model configuration<br/>[model.py]"]
  node_runtime_config["Runtime configuration<br/>[vllm.py]"]
end

subgraph group_cache["KV cache"]
  node_kv_interface["Cache specifications"]
  node_cache_manager["Cache manager"]
  node_cache_layout["Cache layout<br/>[kv_cache_utils.py]"]
  node_ragged_paging["Ragged paging"]
  node_compression["Compression scorers"]
  node_kv_transfer["Shared KV storage"]
end

subgraph group_compute["Model execution"]
  node_attention["Attention backends<br/>[abstract.py]"]
  node_model_layers["Model layers"]
  node_lora["LoRA adapters<br/>[worker_manager.py]"]
  node_distributed["Distributed execution<br/>[parallel_state.py]"]
end

node_client(("Generation client"))

node_client -->|"submits generation"| node_llm_api
node_client -.->|"sends requests"| node_openai_api
node_openai_api -->|"uses protocol"| node_protocol
node_openai_api -.->|"serves through"| node_engine
node_llm_api -.->|"runs generation"| node_engine
node_engine -->|"creates requests"| node_request
node_engine -->|"builds from config"| node_runtime_config
node_runtime_config -->|"configures model"| node_model_config
node_engine -.->|"manages cache"| node_cache_manager
node_cache_manager -->|"uses utilities"| node_cache_layout
node_cache_manager -->|"uses specifications"| node_kv_interface
node_cache_layout -->|"uses specifications"| node_kv_interface
node_ragged_paging -->|"enables retention"| node_compression
node_cache_manager -.->|"allocates paged cache"| node_ragged_paging
node_compression -->|"preserves across turns"| node_request
node_kv_transfer -->|"loads cache"| node_cache_manager
node_engine -->|"supports paged attention"| node_attention
node_engine -.->|"executes model"| node_model_layers
node_engine -->|"manages adapters"| node_lora
node_engine -.->|"uses parallel execution"| node_distributed

click node_llm_api "https://github.com/aiha-lab/tangram/blob/main/vllm/entrypoints/llm.py"
click node_openai_api "https://github.com/aiha-lab/tangram/blob/main/vllm/entrypoints/openai/serving_engine.py"
click node_protocol "https://github.com/aiha-lab/tangram/blob/main/vllm/entrypoints/openai/protocol.py"
click node_engine "https://github.com/aiha-lab/tangram/blob/main/vllm/v1/engine/llm_engine.py"
click node_request "https://github.com/aiha-lab/tangram/blob/main/vllm/v1/request.py"
click node_model_config "https://github.com/aiha-lab/tangram/blob/main/vllm/config/model.py"
click node_runtime_config "https://github.com/aiha-lab/tangram/blob/main/vllm/config/vllm.py"
click node_kv_interface "https://github.com/aiha-lab/tangram/blob/main/vllm/v1/kv_cache_interface.py"
click node_cache_manager "https://github.com/aiha-lab/tangram/blob/main/vllm/v1/core/kv_cache_manager.py"
click node_cache_layout "https://github.com/aiha-lab/tangram/blob/main/vllm/v1/core/kv_cache_utils.py"
click node_kv_transfer "https://github.com/aiha-lab/tangram/blob/main/vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py"
click node_attention "https://github.com/aiha-lab/tangram/blob/main/vllm/attention/backends/abstract.py"
click node_model_layers "https://github.com/aiha-lab/tangram/tree/main/vllm/model_executor/layers"
click node_lora "https://github.com/aiha-lab/tangram/blob/main/vllm/lora/worker_manager.py"
click node_distributed "https://github.com/aiha-lab/tangram/blob/main/vllm/distributed/parallel_state.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_llm_api,node_openai_api,node_protocol,node_client toneBlue
class node_engine,node_request,node_model_config,node_runtime_config toneAmber
class node_kv_interface,node_cache_manager,node_cache_layout,node_ragged_paging,node_compression,node_kv_transfer toneMint
class node_attention,node_model_layers,node_lora,node_distributed toneRose
```

## Components

### Serving interface

- [LLM API](https://github.com/aiha-lab/tangram/blob/main/vllm/entrypoints/llm.py): `vllm/entrypoints/llm.py`
- [OpenAI serving](https://github.com/aiha-lab/tangram/blob/main/vllm/entrypoints/openai/serving_engine.py): `vllm/entrypoints/openai/serving_engine.py`
- [API protocol](https://github.com/aiha-lab/tangram/blob/main/vllm/entrypoints/openai/protocol.py): `vllm/entrypoints/openai/protocol.py`

### Request runtime

- [V1 engine](https://github.com/aiha-lab/tangram/blob/main/vllm/v1/engine/llm_engine.py): `vllm/v1/engine/llm_engine.py`
- [Request state](https://github.com/aiha-lab/tangram/blob/main/vllm/v1/request.py): `vllm/v1/request.py`
- [Model configuration](https://github.com/aiha-lab/tangram/blob/main/vllm/config/model.py): `vllm/config/model.py`
- [Runtime configuration](https://github.com/aiha-lab/tangram/blob/main/vllm/config/vllm.py): `vllm/config/vllm.py`

### KV cache

- [Cache specifications](https://github.com/aiha-lab/tangram/blob/main/vllm/v1/kv_cache_interface.py): `vllm/v1/kv_cache_interface.py`
- [Cache manager](https://github.com/aiha-lab/tangram/blob/main/vllm/v1/core/kv_cache_manager.py): `vllm/v1/core/kv_cache_manager.py`
- [Cache layout](https://github.com/aiha-lab/tangram/blob/main/vllm/v1/core/kv_cache_utils.py): `vllm/v1/core/kv_cache_utils.py`
- **Ragged paging**
- **Compression scorers**
- [Shared KV storage](https://github.com/aiha-lab/tangram/blob/main/vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py): `vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py`

### Model execution

- [Attention backends](https://github.com/aiha-lab/tangram/blob/main/vllm/attention/backends/abstract.py): `vllm/attention/backends/abstract.py`
- [Model layers](https://github.com/aiha-lab/tangram/tree/main/vllm/model_executor/layers): `vllm/model_executor/layers`
- [LoRA adapters](https://github.com/aiha-lab/tangram/blob/main/vllm/lora/worker_manager.py): `vllm/lora/worker_manager.py`
- [Distributed execution](https://github.com/aiha-lab/tangram/blob/main/vllm/distributed/parallel_state.py): `vllm/distributed/parallel_state.py`

### Other components

- **Generation client**

## Connections

- Generation client → LLM API: submits generation
- Generation client → OpenAI serving: sends requests
- OpenAI serving → API protocol: uses protocol (evidence: `vllm/entrypoints/openai/serving_engine.py`)
- OpenAI serving → V1 engine: serves through
- LLM API → V1 engine: runs generation
- V1 engine → Request state: creates requests (evidence: `vllm/v1/engine/llm_engine.py`)
- V1 engine → Runtime configuration: builds from config (evidence: `vllm/v1/engine/llm_engine.py`)
- Runtime configuration → Model configuration: configures model (evidence: `vllm/config/vllm.py`)
- V1 engine → Cache manager: manages cache
- Cache manager → Cache layout: uses utilities (evidence: `vllm/v1/core/kv_cache_manager.py`)
- Cache manager → Cache specifications: uses specifications (evidence: `vllm/v1/core/kv_cache_manager.py`)
- Cache layout → Cache specifications: uses specifications (evidence: `vllm/v1/core/kv_cache_utils.py`)
- Ragged paging → Compression scorers: enables retention
- Cache manager → Ragged paging: allocates paged cache
- Compression scorers → Request state: preserves across turns
- Shared KV storage → Cache manager: loads cache (evidence: `vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py`)
- V1 engine → Attention backends: supports paged attention
- V1 engine → Model layers: executes model
- V1 engine → LoRA adapters: manages adapters (evidence: `vllm/v1/engine/llm_engine.py`)
- V1 engine → Distributed execution: uses parallel execution (evidence: `vllm/v1/engine/llm_engine.py`)
