# pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane architecture

> Architecture diagram of the GitHub repository pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane
- Repository: https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane

## Overview

This repository implements a paged-KV-cache LLM inference service. A client submits a completion request; the API tokenizes it and hands a sequence to the engine, which schedules prefill/decode work, runs model steps with paged attention, and returns generated text. The main internal layers are scheduling and sequence state, cache allocation/storage strategies, and native page allocation. The tree and excerpts also show a separate step-generation endpoint and metrics endpoint. Some implementation details are unsampled, so uncertain links are omitted or marked unverified.

Main entry points: [`app/engine/llm_engine.py`](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/engine/llm_engine.py) (LLM engine), [`app/api/server.py`](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/api/server.py) (HTTP endpoints), [`app/memory/manager.py`](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/memory/manager.py) (KV cache manager), [`app/model/paged_attention.py`](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/model/paged_attention.py) (Paged attention)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Service API"]
  node_server["HTTP endpoints<br/>[server.py]"]
  node_config["Runtime config<br/>[config.py]"]
end

subgraph group_runtime["Inference Runtime"]
  node_engine["LLM engine<br/>[llm_engine.py]"]
  node_sequence["Request sequence<br/>[sequence.py]"]
  node_scheduler["Batch scheduler<br/>[scheduler.py]"]
  node_model["Model inference<br/>[server.py]"]
  node_model_client["Remote model client<br/>[client.py]"]
end

subgraph group_cache["Paged KV Cache"]
  node_attention["Paged attention<br/>[paged_attention.py]"]
  node_attention_backends["Attention backends"]
  node_cache_manager["KV cache manager<br/>[manager.py]"]
  node_memory_strategies["Cache strategies<br/>[strategies.py]"]
end

subgraph group_native["Native Allocation"]
  node_allocator_binding["Allocator bindings<br/>[bindings.cpp]"]
  node_page_allocator["Page allocator<br/>[page_allocator.hpp]"]
  node_kernel_api["Paged kernel API<br/>[paged_kernels.h]"]
  node_kernels["Paged kernels<br/>[paged_kernels.cpp]"]
end

node_caller(("API caller"))

node_caller -->|"submits request"| node_server
node_server -->|"adds sequence"| node_engine
node_server -->|"runs generation"| node_model
node_engine -->|"creates state"| node_sequence
node_engine -->|"adds and schedules"| node_scheduler
node_scheduler -->|"tracks work"| node_sequence
node_engine -->|"writes KV cache"| node_cache_manager
node_engine -->|"runs paged forward"| node_attention
node_engine -->|"requests model step"| node_model_client
node_attention -->|"maps cache slots"| node_cache_manager
node_cache_manager -->|"allocates cache"| node_memory_strategies
node_cache_manager -->|"uses page allocator"| node_allocator_binding
node_allocator_binding -->|"binds allocator"| node_page_allocator
node_allocator_binding -->|"imports kernel API"| node_kernel_api
node_kernels -->|"implements API"| node_kernel_api
node_attention -.->|"selects backend"| node_attention_backends
node_server -->|"uses defaults"| node_config

click node_server "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/api/server.py"
click node_config "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/config.py"
click node_engine "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/engine/llm_engine.py"
click node_sequence "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/scheduler/sequence.py"
click node_scheduler "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/scheduler/scheduler.py"
click node_model "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/api/server.py"
click node_model_client "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/model/client.py"
click node_attention "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/model/paged_attention.py"
click node_attention_backends "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/model/attention_backends.py"
click node_cache_manager "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/memory/manager.py"
click node_memory_strategies "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/memory/strategies.py"
click node_allocator_binding "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/csrc/src/bindings.cpp"
click node_page_allocator "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/csrc/include/page_allocator.hpp"
click node_kernel_api "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/csrc/include/paged_kernels.h"
click node_kernels "https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/csrc/src/paged_kernels.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server,node_config toneBlue
class node_engine,node_sequence,node_scheduler,node_model,node_model_client toneAmber
class node_attention,node_attention_backends,node_cache_manager,node_memory_strategies,node_caller toneMint
class node_allocator_binding,node_page_allocator,node_kernel_api,node_kernels toneRose
```

## Components

### Service API

- [HTTP endpoints](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/api/server.py): `app/api/server.py`
- [Runtime config](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/config.py): `app/config.py`

### Inference Runtime

- [LLM engine](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/engine/llm_engine.py): `app/engine/llm_engine.py`
- [Request sequence](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/scheduler/sequence.py): `app/scheduler/sequence.py`
- [Batch scheduler](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/scheduler/scheduler.py): `app/scheduler/scheduler.py`
- [Model inference](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/api/server.py): `app/api/server.py`
- [Remote model client](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/model/client.py): `app/model/client.py`

### Paged KV Cache

- [Paged attention](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/model/paged_attention.py): `app/model/paged_attention.py`
- [Attention backends](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/model/attention_backends.py): `app/model/attention_backends.py`
- [KV cache manager](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/memory/manager.py): `app/memory/manager.py`
- [Cache strategies](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/app/memory/strategies.py): `app/memory/strategies.py`

### Native Allocation

- [Allocator bindings](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/csrc/src/bindings.cpp): `csrc/src/bindings.cpp`
- [Page allocator](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/csrc/include/page_allocator.hpp): `csrc/include/page_allocator.hpp`
- [Paged kernel API](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/csrc/include/paged_kernels.h): `csrc/include/paged_kernels.h`
- [Paged kernels](https://github.com/pakanor/low-latency-distributed-llm-serving-infrastructure-custom-k8s-control-plane/blob/main/csrc/src/paged_kernels.cpp): `csrc/src/paged_kernels.cpp`

### Other components

- **API caller**

## Connections

- API caller → HTTP endpoints: submits request (evidence: `app/api/server.py`)
- HTTP endpoints → LLM engine: adds sequence (evidence: `app/api/server.py`)
- HTTP endpoints → Model inference: runs generation (evidence: `app/api/server.py`)
- LLM engine → Request sequence: creates state (evidence: `app/engine/llm_engine.py`)
- LLM engine → Batch scheduler: adds and schedules (evidence: `app/engine/llm_engine.py`)
- Batch scheduler → Request sequence: tracks work (evidence: `app/scheduler/scheduler.py`)
- LLM engine → KV cache manager: writes KV cache (evidence: `app/engine/llm_engine.py`)
- LLM engine → Paged attention: runs paged forward (evidence: `app/engine/llm_engine.py`)
- LLM engine → Remote model client: requests model step (evidence: `app/engine/llm_engine.py`)
- Paged attention → KV cache manager: maps cache slots (evidence: `app/model/paged_attention.py`)
- KV cache manager → Cache strategies: allocates cache (evidence: `app/memory/manager.py`)
- KV cache manager → Allocator bindings: uses page allocator (evidence: `app/memory/manager.py`)
- Allocator bindings → Page allocator: binds allocator (evidence: `csrc/src/bindings.cpp`)
- Allocator bindings → Paged kernel API: imports kernel API (evidence: `csrc/src/bindings.cpp`)
- Paged kernels → Paged kernel API: implements API (evidence: `csrc/src/paged_kernels.cpp`)
- Paged attention → Attention backends: selects backend (evidence: `app/model/paged_attention.py`)
- HTTP endpoints → Runtime config: uses defaults (evidence: `app/api/server.py`)