# gavin-d26/corellm architecture

> Architecture diagram of the GitHub repository gavin-d26/corellm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gavin-d26/corellm
- Repository: https://github.com/gavin-d26/corellm

## Overview

1. **What this repo is**
   CoreLLM is a minimal LLM inference engine, not a training stack. Its architecture is centered on fast serving of a single supported family, Llama 3.1, with an emphasis on continuous batching, paged KV cache management, and multi-GPU tensor parallelism.

2. **Top-level runtime modes**
   There are two entry paths:
   - **Offline Python API** via `EngineNode`, used for batch inference directly from code.
   - **OpenAI-compatible server** via `corellm serve`, which exposes `/v1/completions` and `/v1/chat/completions` for online requests.
   Both paths converge on the same engine and worker runtime.

3. **Request ingestion and client boundary**
   The server-side boundary is in `corellm/openai_server`. It accepts HTTP requests, maps them to OpenAI-style schemas, tokenizes prompts, and forwards work to the engine client. The client layer is the boundary between API concerns and model execution; it keeps tokenization/detokenization and request shaping off the GPU hot path.

4. **Main engine control plane**
   `corellm/engine/engine.py` is the orchestrator. It owns:
   - the serving loop,
   - scheduler decisions,
   - sequence state,
   - KV-cache bookkeeping,
   - and coordination with GPU workers.
   The engine is the “control plane” of the system; it does not itself contain the model math.

5. **Worker / model execution plane**
   GPU workers run model forward passes and own the model shards. The repo’s architecture description indicates a controller/worker split with:
   - **ZMQ** for control messages,
   - **NCCL** for tensor-parallel collectives.
   This keeps HTTP, scheduling, and token management out of the worker process. The worker side is responsible for the actual forward compute and returns outputs/results to the engine.

6. **Scheduling and continuous batching**
   `corellm/engine/scheduler.py` determines the next batch composition. The system is built around a single mixed forward pass that can contain both:
   - **prefill tokens** from new prompts, and
   - **decode tokens** from in-flight sequences.
   The scheduler is decode-prioritized and chunks prefill to respect batching limits. This is the core throughput-oriented control loop in the architecture.

7. **Sequence and KV-cache state**
   `corellm/engine/sequence.py` and `corellm/engine/kv_cache/` hold per-request state and KV allocation logic. There are two cache allocators:
   - **naive**
   - **radix-tree-based prefix caching**
   The KV cache is a major subsystem because it enables reuse of prompt prefixes and is tightly coupled to batch scheduling and memory layout.

8. **Model graph and weight loading**
   `corellm/engine/model_graph/` is the bridge from model definition to executable graph:
   - `executor.py` runs the model graph on the worker side,
   - `weight_loader.py` materializes pretrained weights into the model layout.
   This layer separates model structure from runtime execution and is where checkpoint loading meets the inference graph.

9. **Model implementation and reusable layers**
   `corellm/models/` defines supported model wrappers and templates, currently centered on Llama 3.1. `corellm/layers/` contains the reusable neural network primitives: attention, linear, embeddings, normalization, and activations. These are the semantic model components that the executor assembles into a forward pass.

10. **Custom GPU kernels**
   `corellm/kernels/` and `corellm/engine/kv_cache/` are where the performance-critical CUDA/Triton pieces live. The repo explicitly includes:
   - **paged attention**
   - **gather**
   - device-specific kernel configs
   The README emphasizes a custom paged-attention kernel that handles mixed prefill/decode batches and arbitrary sequence lengths. This is the low-level compute boundary beneath the model graph.

11. **Memory layout and parallelism**
   A key architectural decision is the **single flat GPU allocation** per rank: model weights occupy the front of the slab, and the remainder is carved into KV blocks. The amount of cache is derived from available device memory. On multi-GPU setups, the model is tensor-parallel, with sharded attention and fused column/row-parallel linears. This makes memory placement and collectives first-class architectural concerns.

12. **Serving and benchmarking as separate subsystems**
   `corellm/openai_server/` provides the public API surface, while `benchmarks/` contains offline throughput, latency, serving, and kernel microbenchmarks. These benchmarking scripts are not part of the serving hot path, but they document the intended usage patterns and performance envelope of the engine. The `results/` directory stores recorded benchmark outputs.

13. **Tests and validation boundaries**
   The tests focus on integration behavior rather than unit-level numerical checks:
   - `tests/test_chat_batch.py`
   - `tests/test_server_requests.py`
   This suggests the most important correctness boundaries are request batching and server/API behavior, not isolated math kernels.

14. **Overall data flow to draw in a diagram**
   A request typically flows as:
   **HTTP/OpenAI request → server schemas → tokenization/client → engine scheduler/sequence state → worker forward pass → custom kernels on GPU → sampled tokens/results → detokenization → response streaming or return**.
   In offline mode, the same flow exists but without the HTTP/server front end. The main architectural boundary is between the control plane in the engine/server processes and the compute plane in GPU worker processes.

Main entry points: [`corellm/engine/engine.py`](https://github.com/gavin-d26/corellm/blob/main/corellm/engine/engine.py) (Engine), [`corellm/engine/kv_cache`](https://github.com/gavin-d26/corellm/tree/main/corellm/engine/kv_cache) (KV Cache), [`benchmarks`](https://github.com/gavin-d26/corellm/tree/main/benchmarks) (Benchmarks), [`corellm/openai_server/server.py`](https://github.com/gavin-d26/corellm/blob/main/corellm/openai_server/server.py) (Server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_api["API & CLI"]
  node_node_cli["CLI<br/>entrypoint<br/>[cli.py]"]
  node_node_server["Server<br/>http api<br/>[server.py]"]
  node_node_schemas["Schemas<br/>request models<br/>[schemas.py]"]
  node_node_engine_client["Engine Client<br/>control client<br/>[engine_client.py]"]
end

subgraph group_group_engine["Engine Control"]
  node_node_engine["Engine<br/>orchestrator<br/>[engine.py]"]
  node_node_scheduler["Scheduler<br/>batch planner<br/>[scheduler.py]"]
  node_node_sequence["Sequence State<br/>request state<br/>[sequence.py]"]
  node_node_kv_cache[("KV Cache<br/>memory manager")]
  node_node_naive_alloc["Naive Alloc<br/>allocator"]
  node_node_radix_alloc["Radix Alloc<br/>prefix cache"]
end

subgraph group_group_worker["Model Runtime"]
  node_node_comms["Comms<br/>ipc & collectives<br/>[comms.py]"]
  node_node_executor["Executor<br/>model runner<br/>[executor.py]"]
  node_node_weight_loader["Weight Loader<br/>checkpoint loader<br/>[weight_loader.py]"]
  node_node_kernels{{"Kernels"}}
end

subgraph group_group_model["Model Stack"]
  node_node_model["Llama 3.1<br/>model wrapper<br/>[llama3.py]"]
  node_node_template["Template<br/>[template.py]"]
  node_node_layers["Layers<br/>nn primitives"]
end

subgraph group_group_perf["Benchmarks"]
  node_node_bench["Benchmarks<br/>perf suite"]
  node_node_results["Results<br/>outputs"]
end

node_node_cli -->|"serve"| node_node_server
node_node_cli -->|"offline api"| node_node_engine
node_node_server -->|"parse"| node_node_schemas
node_node_schemas -->|"shape"| node_node_engine_client
node_node_engine_client -->|"submit"| node_node_engine
node_node_engine -->|"plan"| node_node_scheduler
node_node_engine -->|"track"| node_node_sequence
node_node_engine -->|"allocate"| node_node_kv_cache
node_node_scheduler -->|"read state"| node_node_sequence
node_node_scheduler -->|"place tokens"| node_node_kv_cache
node_node_kv_cache -->|"strategy"| node_node_naive_alloc
node_node_kv_cache -->|"strategy"| node_node_radix_alloc
node_node_engine -->|"dispatch"| node_node_comms
node_node_comms -->|"run"| node_node_executor
node_node_executor -->|"load"| node_node_weight_loader
node_node_weight_loader -->|"materialize"| node_node_model
node_node_model -->|"format"| node_node_template
node_node_model -->|"compose"| node_node_layers
node_node_executor -->|"invoke"| node_node_kernels
node_node_kernels -->|"collectives"| node_node_comms
node_node_bench -->|"measure"| node_node_kernels
node_node_bench -->|"stress"| node_node_engine
node_node_bench -->|"record"| node_node_results
node_node_server -->|"request flow"| node_node_engine_client

click node_node_cli "https://github.com/gavin-d26/corellm/blob/main/corellm/cli.py"
click node_node_server "https://github.com/gavin-d26/corellm/blob/main/corellm/openai_server/server.py"
click node_node_schemas "https://github.com/gavin-d26/corellm/blob/main/corellm/openai_server/schemas.py"
click node_node_engine_client "https://github.com/gavin-d26/corellm/blob/main/corellm/engine/engine_client.py"
click node_node_engine "https://github.com/gavin-d26/corellm/blob/main/corellm/engine/engine.py"
click node_node_scheduler "https://github.com/gavin-d26/corellm/blob/main/corellm/engine/scheduler.py"
click node_node_sequence "https://github.com/gavin-d26/corellm/blob/main/corellm/engine/sequence.py"
click node_node_kv_cache "https://github.com/gavin-d26/corellm/tree/main/corellm/engine/kv_cache"
click node_node_naive_alloc "https://github.com/gavin-d26/corellm/blob/main/corellm/engine/kv_cache/naive_cache_alloc.py"
click node_node_radix_alloc "https://github.com/gavin-d26/corellm/blob/main/corellm/engine/kv_cache/radix_cache_alloc.py"
click node_node_comms "https://github.com/gavin-d26/corellm/blob/main/corellm/comms.py"
click node_node_executor "https://github.com/gavin-d26/corellm/blob/main/corellm/engine/model_graph/executor.py"
click node_node_weight_loader "https://github.com/gavin-d26/corellm/blob/main/corellm/engine/model_graph/weight_loader.py"
click node_node_model "https://github.com/gavin-d26/corellm/blob/main/corellm/models/llama3.py"
click node_node_template "https://github.com/gavin-d26/corellm/blob/main/corellm/models/template.py"
click node_node_layers "https://github.com/gavin-d26/corellm/tree/main/corellm/layers"
click node_node_kernels "https://github.com/gavin-d26/corellm/tree/main/corellm/kernels"
click node_node_bench "https://github.com/gavin-d26/corellm/tree/main/benchmarks"
click node_node_results "https://github.com/gavin-d26/corellm/tree/main/results"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_cli,node_node_server,node_node_schemas,node_node_engine_client toneBlue
class node_node_engine,node_node_scheduler,node_node_sequence,node_node_kv_cache,node_node_naive_alloc,node_node_radix_alloc toneAmber
class node_node_comms,node_node_executor,node_node_weight_loader,node_node_kernels toneMint
class node_node_model,node_node_template,node_node_layers toneRose
class node_node_bench,node_node_results toneIndigo
```

## Components

### API & CLI

Public entry points and request shaping

- [CLI](https://github.com/gavin-d26/corellm/blob/main/corellm/cli.py): entrypoint · Starts server or offline inference modes · `corellm/cli.py`
- [Server](https://github.com/gavin-d26/corellm/blob/main/corellm/openai_server/server.py): http api · OpenAI-compatible serving front end · `corellm/openai_server/server.py`
- [Schemas](https://github.com/gavin-d26/corellm/blob/main/corellm/openai_server/schemas.py): request models · OpenAI request and response schema layer · `corellm/openai_server/schemas.py`
- [Engine Client](https://github.com/gavin-d26/corellm/blob/main/corellm/engine/engine_client.py): control client · Boundary between API handling and engine execution · `corellm/engine/engine_client.py`

### Engine Control

Scheduling, sequence state, and KV-cache coordination

- [Engine](https://github.com/gavin-d26/corellm/blob/main/corellm/engine/engine.py): orchestrator · Serving loop and control-plane coordinator · `corellm/engine/engine.py`
- [Scheduler](https://github.com/gavin-d26/corellm/blob/main/corellm/engine/scheduler.py): batch planner · Builds mixed prefill/decode batches · `corellm/engine/scheduler.py`
- [Sequence State](https://github.com/gavin-d26/corellm/blob/main/corellm/engine/sequence.py): request state · Per-request lifecycle and token state · `corellm/engine/sequence.py`
- [KV Cache](https://github.com/gavin-d26/corellm/tree/main/corellm/engine/kv_cache): memory manager · Paged KV allocation and prefix reuse · `corellm/engine/kv_cache`
- [Naive Alloc](https://github.com/gavin-d26/corellm/blob/main/corellm/engine/kv_cache/naive_cache_alloc.py): allocator · Simple KV allocation strategy · `corellm/engine/kv_cache/naive_cache_alloc.py`
- [Radix Alloc](https://github.com/gavin-d26/corellm/blob/main/corellm/engine/kv_cache/radix_cache_alloc.py): prefix cache · Radix-tree KV reuse for shared prefixes · `corellm/engine/kv_cache/radix_cache_alloc.py`

### Model Runtime

Model execution and GPU-side compute

- [Comms](https://github.com/gavin-d26/corellm/blob/main/corellm/comms.py): ipc & collectives · Control messaging and distributed transport · `corellm/comms.py`
- [Executor](https://github.com/gavin-d26/corellm/blob/main/corellm/engine/model_graph/executor.py): model runner · Runs the model graph on worker ranks · `corellm/engine/model_graph/executor.py`
- [Weight Loader](https://github.com/gavin-d26/corellm/blob/main/corellm/engine/model_graph/weight_loader.py): checkpoint loader · Materializes pretrained weights into runtime layout · `corellm/engine/model_graph/weight_loader.py`
- [Kernels](https://github.com/gavin-d26/corellm/tree/main/corellm/kernels): gpu kernels · Paged attention, gather, and device configs · `corellm/kernels`

### Model Stack

Supported model wrappers and reusable layers

- [Llama 3.1](https://github.com/gavin-d26/corellm/blob/main/corellm/models/llama3.py): model wrapper · Supported model family and template integration · `corellm/models/llama3.py`
- [Template](https://github.com/gavin-d26/corellm/blob/main/corellm/models/template.py): prompt template · Model prompt formatting and adaptation · `corellm/models/template.py`
- [Layers](https://github.com/gavin-d26/corellm/tree/main/corellm/layers): nn primitives · Reusable attention, linear, embedding, and norm blocks · `corellm/layers`

### Benchmarks

Offline performance evaluation and recorded outputs

- [Benchmarks](https://github.com/gavin-d26/corellm/tree/main/benchmarks): perf suite · Throughput, latency, serving, and kernel tests · `benchmarks`
- [Results](https://github.com/gavin-d26/corellm/tree/main/results): outputs · Captured benchmark outputs and comparisons · `results`

## Connections

- CLI → Server: serve · CLI launches the OpenAI-compatible server
- CLI → Engine: offline api · CLI also reaches the offline engine path
- Server → Schemas: parse · HTTP requests are mapped into OpenAI-style schemas
- Schemas → Engine Client: shape · Validated requests are forwarded through the engine client
- Engine Client → Engine: submit · Client hands work to the engine control plane
- Engine → Scheduler: plan · Engine asks scheduler for next batch composition
- Engine → Sequence State: track · Engine maintains per-request sequence state
- Engine → KV Cache: allocate · Engine coordinates KV-cache bookkeeping
- Scheduler → Sequence State: read state · Batch decisions depend on sequence progress
- Scheduler → KV Cache: place tokens · Scheduler respects cache and batching limits
- KV Cache → Naive Alloc: strategy · One cache backend is a simple allocator
- KV Cache → Radix Alloc: strategy · Prefix reuse uses a radix-tree allocator
- Engine → Comms: dispatch · Engine uses control messaging toward worker ranks
- Comms → Executor: run · Worker side executes model forward passes
- Executor → Weight Loader: load · Executor consumes loaded model weights
- Weight Loader → Llama 3.1: materialize · Weights are arranged according to the supported model graph
- Llama 3.1 → Template: format · Model wrapper uses prompt templates
- Llama 3.1 → Layers: compose · Model is built from reusable neural layers
- Executor → Kernels: invoke · Execution reaches custom GPU kernels
- Kernels → Comms: collectives · Tensor-parallel execution relies on distributed comms
- Benchmarks → Kernels: measure · Kernel benchmarks isolate compute performance
- Benchmarks → Engine: stress · Serving and throughput benchmarks exercise the engine path
- Benchmarks → Results: record · Benchmark runs produce stored comparison outputs
- Server → Engine Client: request flow · Server forwards live requests through the engine client