How pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane works
This repository implements a paged-KV-cache LLM inference service. A client submits a completion request; the API tokenizes it and hands a sequence to the engine, which schedules prefill/decode work, runs model steps with paged attention, and returns generated text. The main internal layers are scheduling and sequence state, cache allocation/storage strategies, and native page allocation. The tree and excerpts also show a separate step-generation endpoint and metrics endpoint. Some implementation details are unsampled, so uncertain links are omitted or marked unverified.
Main entry points: llm_engine.py LLM engine · server.py HTTP endpoints · manager.py KV cache manager · paged_attention.py Paged attention
Components 16
Service API
- HTTP endpoints
app/api/server.py - Runtime config
app/config.py
Inference Runtime
- LLM engine
app/engine/llm_engine.py - Request sequence
app/scheduler/sequence.py - Batch scheduler
app/scheduler/scheduler.py - Model inference
app/api/server.py - Remote model client
app/model/client.py
Paged KV Cache
- Paged attention
app/model/paged_attention.py - Attention backends
app/model/attention_backends.py - KV cache manager
app/memory/manager.py - Cache strategies
app/memory/strategies.py
Native Allocation
- Allocator bindings
csrc/src/bindings.cpp - Page allocator
csrc/include/page_allocator.hpp - Paged kernel API
csrc/include/paged_kernels.h - Paged kernels
csrc/src/paged_kernels.cpp
Other components
- API caller
Connections 17
- API caller → HTTP endpoints: submits request
app/api/server.py - HTTP endpoints → LLM engine: adds sequence
app/api/server.py - HTTP endpoints → Model inference: runs generation
app/api/server.py - LLM engine → Request sequence: creates state
app/engine/llm_engine.py - LLM engine → Batch scheduler: adds and schedules
app/engine/llm_engine.py - Batch scheduler → Request sequence: tracks work
app/scheduler/scheduler.py - LLM engine → KV cache manager: writes KV cache
app/engine/llm_engine.py - LLM engine → Paged attention: runs paged forward
app/engine/llm_engine.py - LLM engine → Remote model client: requests model step
app/engine/llm_engine.py - Paged attention → KV cache manager: maps cache slots
app/model/paged_attention.py - KV cache manager → Cache strategies: allocates cache
app/memory/manager.py - KV cache manager → Allocator bindings: uses page allocator
app/memory/manager.py - Allocator bindings → Page allocator: binds allocator
csrc/src/bindings.cpp - Allocator bindings → Paged kernel API: imports kernel API
csrc/src/bindings.cpp - Paged kernels → Paged kernel API: implements API
csrc/src/paged_kernels.cpp - Paged attention → Attention backends: selects backend
app/model/paged_attention.py - HTTP endpoints → Runtime config: uses defaults
app/api/server.py