How aiha-lab/tangram works
Tangram extends vLLM with non-uniform KV-cache compression for multi-turn LLM serving. The main path is a generation request entering the serving interface, reaching the engine, and using paged KV-cache management so compressed state can be reused across turns. README documentation describes compression scorers, budget scopes, ragged paging, memory reclamation, and compatibility with vLLM serving modes. The tree does not include the documented compression implementation paths, so those are represented as an unlocated subsystem; runtime links beyond the shown excerpts are limited to documented relationships.
Main entry points: llm_engine.py V1 engine · kv_cache_manager.py Cache manager · serving_engine.py OpenAI serving · llm.py LLM API
Components 18
Serving interface
- LLM API
vllm/entrypoints/llm.py - OpenAI serving
vllm/entrypoints/openai/serving_engine.py - API protocol
vllm/entrypoints/openai/protocol.py
Request runtime
- V1 engine
vllm/v1/engine/llm_engine.py - Request state
vllm/v1/request.py - Model configuration
vllm/config/model.py - Runtime configuration
vllm/config/vllm.py
KV cache
- Cache specifications
vllm/v1/kv_cache_interface.py - Cache manager
vllm/v1/core/kv_cache_manager.py - Cache layout
vllm/v1/core/kv_cache_utils.py - Ragged paging
- Compression scorers
- Shared KV storage
vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py
Model execution
- Attention backends
vllm/attention/backends/abstract.py - Model layers
vllm/model_executor/layers - LoRA adapters
vllm/lora/worker_manager.py - Distributed execution
vllm/distributed/parallel_state.py
Other components
- Generation client
Connections 20
- Generation client → LLM API: submits generation
- Generation client → OpenAI serving: sends requests
- OpenAI serving → API protocol: uses protocol
vllm/entrypoints/openai/serving_engine.py - OpenAI serving → V1 engine: serves through
- LLM API → V1 engine: runs generation
- V1 engine → Request state: creates requests
vllm/v1/engine/llm_engine.py - V1 engine → Runtime configuration: builds from config
vllm/v1/engine/llm_engine.py - Runtime configuration → Model configuration: configures model
vllm/config/vllm.py - V1 engine → Cache manager: manages cache
- Cache manager → Cache layout: uses utilities
vllm/v1/core/kv_cache_manager.py - Cache manager → Cache specifications: uses specifications
vllm/v1/core/kv_cache_manager.py - Cache layout → Cache specifications: uses specifications
vllm/v1/core/kv_cache_utils.py - Ragged paging → Compression scorers: enables retention
- Cache manager → Ragged paging: allocates paged cache
- Compression scorers → Request state: preserves across turns
- Shared KV storage → Cache manager: loads cache
vllm/distributed/kv_transfer/kv_connector/v1/shared_storage_connector.py - V1 engine → Attention backends: supports paged attention
- V1 engine → Model layers: executes model
- V1 engine → LoRA adapters: manages adapters
vllm/v1/engine/llm_engine.py - V1 engine → Distributed execution: uses parallel execution
vllm/v1/engine/llm_engine.py