# binbinao/cllm_inference architecture

> Architecture diagram of the GitHub repository binbinao/cllm_inference, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/binbinao/cllm_inference
- Repository: https://github.com/binbinao/cllm_inference

## Overview

cllm_inference is a C++20 CPU GGUF inference engine with two user-facing modes: local CLI generation and an OpenAI-compatible HTTP/SSE service. Both load a model, tokenize a prompt, run autoregressive Transformer inference, sample tokens, and decode output. The diagram separates model parsing, inference kernels, and serving; the README documents additional GGUF, quantization, tokenizer, and API capabilities beyond the sampled excerpts. Edges are limited to excerpted calls or explicit README relationships; several implementation details were not sampled.

Main entry points: [`main.cpp`](https://github.com/binbinao/cllm_inference/blob/main/main.cpp) (CLI orchestration), [`src/server/http_server.cpp`](https://github.com/binbinao/cllm_inference/blob/main/src/server/http_server.cpp) (HTTP API routes), [`src/core/engine.cpp`](https://github.com/binbinao/cllm_inference/blob/main/src/core/engine.cpp) (Transformer engine), [`src/core/tokenizer.cpp`](https://github.com/binbinao/cllm_inference/blob/main/src/core/tokenizer.cpp) (Tokenizer and templates)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry points"]
  node_cli["CLI orchestration<br/>[main.cpp]"]
  node_cli_output["CLI output<br/>[main.cpp]"]
end

subgraph group_model["Model and tokens"]
  node_loader["GGUF loader<br/>[gguf.cpp]"]
  node_tensor["Mapped tensors<br/>[tensor.hpp]"]
  node_quant["Weight dequantization<br/>[quant.cpp]"]
  node_tokenizer["Tokenizer and templates<br/>[tokenizer.cpp]"]
end

subgraph group_inference["Inference runtime"]
  node_engine["Transformer engine<br/>[engine.cpp]"]
  node_sampler["Token sampler<br/>[sampler.cpp]"]
  node_threadpool["Thread pool<br/>[thread_pool.cpp]"]
end

subgraph group_serving["HTTP serving"]
  node_http["HTTP server<br/>[http_server.cpp]"]
  node_api["HTTP API routes<br/>[http_server.cpp]"]
  node_http_response["JSON / SSE response<br/>[http_server.cpp]"]
end

node_user(("User"))
node_model_file["GGUF model file"]
node_http_client(("HTTP client"))

node_user -->|"provides options"| node_cli
node_cli -->|"loads model"| node_loader
node_model_file -->|"reads GGUF"| node_loader
node_loader -->|"maps tensors"| node_tensor
node_tensor -->|"dequantizes with"| node_quant
node_cli -->|"initializes engine"| node_engine
node_cli -->|"initializes tokenizer"| node_tokenizer
node_engine -->|"dequantizes weights"| node_tensor
node_engine -->|"parallelizes matmul"| node_threadpool
node_cli -->|"creates pool"| node_threadpool
node_cli -->|"encodes prompt"| node_tokenizer
node_cli -->|"forwards tokens"| node_engine
node_cli -->|"samples logits"| node_sampler
node_cli -->|"decodes tokens"| node_tokenizer
node_cli -->|"prints generation"| node_cli_output
node_user -->|"sends request"| node_http_client
node_http_client -->|"connects"| node_http
node_cli -->|"starts service"| node_http
node_http -->|"dispatches request"| node_api
node_api -->|"encodes prompt"| node_tokenizer
node_api -->|"forwards tokens"| node_engine
node_api -->|"samples logits"| node_sampler
node_api -->|"decodes tokens"| node_tokenizer
node_api -->|"formats output"| node_http_response
node_http_response -->|"returns response"| node_http_client

click node_cli "https://github.com/binbinao/cllm_inference/blob/main/main.cpp"
click node_loader "https://github.com/binbinao/cllm_inference/blob/main/src/core/gguf.cpp"
click node_tensor "https://github.com/binbinao/cllm_inference/blob/main/include/cllm/core/tensor.hpp"
click node_quant "https://github.com/binbinao/cllm_inference/blob/main/src/core/quant.cpp"
click node_tokenizer "https://github.com/binbinao/cllm_inference/blob/main/src/core/tokenizer.cpp"
click node_engine "https://github.com/binbinao/cllm_inference/blob/main/src/core/engine.cpp"
click node_sampler "https://github.com/binbinao/cllm_inference/blob/main/src/core/sampler.cpp"
click node_threadpool "https://github.com/binbinao/cllm_inference/blob/main/src/core/thread_pool.cpp"
click node_http "https://github.com/binbinao/cllm_inference/blob/main/src/server/http_server.cpp"
click node_api "https://github.com/binbinao/cllm_inference/blob/main/src/server/http_server.cpp"
click node_cli_output "https://github.com/binbinao/cllm_inference/blob/main/main.cpp"
click node_http_response "https://github.com/binbinao/cllm_inference/blob/main/src/server/http_server.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_cli_output,node_user,node_http_client toneBlue
class node_loader,node_tensor,node_quant,node_tokenizer toneAmber
class node_engine,node_sampler,node_threadpool toneMint
class node_http,node_api,node_http_response toneRose
class node_model_file toneIndigo
```

## Components

### Entry points

- [CLI orchestration](https://github.com/binbinao/cllm_inference/blob/main/main.cpp): `main.cpp`
- [CLI output](https://github.com/binbinao/cllm_inference/blob/main/main.cpp): `main.cpp`

### Model and tokens

- [GGUF loader](https://github.com/binbinao/cllm_inference/blob/main/src/core/gguf.cpp): `src/core/gguf.cpp`
- [Mapped tensors](https://github.com/binbinao/cllm_inference/blob/main/include/cllm/core/tensor.hpp): `include/cllm/core/tensor.hpp`
- [Weight dequantization](https://github.com/binbinao/cllm_inference/blob/main/src/core/quant.cpp): `src/core/quant.cpp`
- [Tokenizer and templates](https://github.com/binbinao/cllm_inference/blob/main/src/core/tokenizer.cpp): `src/core/tokenizer.cpp`

### Inference runtime

- [Transformer engine](https://github.com/binbinao/cllm_inference/blob/main/src/core/engine.cpp): `src/core/engine.cpp`
- [Token sampler](https://github.com/binbinao/cllm_inference/blob/main/src/core/sampler.cpp): `src/core/sampler.cpp`
- [Thread pool](https://github.com/binbinao/cllm_inference/blob/main/src/core/thread_pool.cpp): `src/core/thread_pool.cpp`

### HTTP serving

- [HTTP server](https://github.com/binbinao/cllm_inference/blob/main/src/server/http_server.cpp): `src/server/http_server.cpp`
- [HTTP API routes](https://github.com/binbinao/cllm_inference/blob/main/src/server/http_server.cpp): `src/server/http_server.cpp`
- [JSON / SSE response](https://github.com/binbinao/cllm_inference/blob/main/src/server/http_server.cpp): `src/server/http_server.cpp`

### Other components

- **User**
- **GGUF model file**
- **HTTP client**

## Connections

- User → CLI orchestration: provides options (evidence: `README.md`)
- CLI orchestration → GGUF loader: loads model (evidence: `main.cpp`)
- GGUF model file → GGUF loader: reads GGUF (evidence: `README.md`)
- GGUF loader → Mapped tensors: maps tensors (evidence: `README.md`)
- Mapped tensors → Weight dequantization: dequantizes with (evidence: `include/cllm/core/tensor.hpp`)
- CLI orchestration → Transformer engine: initializes engine (evidence: `main.cpp`)
- CLI orchestration → Tokenizer and templates: initializes tokenizer (evidence: `main.cpp`)
- Transformer engine → Mapped tensors: dequantizes weights (evidence: `src/core/engine.cpp`)
- Transformer engine → Thread pool: parallelizes matmul (evidence: `src/core/engine.cpp`)
- CLI orchestration → Thread pool: creates pool (evidence: `main.cpp`)
- CLI orchestration → Tokenizer and templates: encodes prompt (evidence: `main.cpp`)
- CLI orchestration → Transformer engine: forwards tokens (evidence: `main.cpp`)
- CLI orchestration → Token sampler: samples logits (evidence: `main.cpp`)
- CLI orchestration → Tokenizer and templates: decodes tokens (evidence: `main.cpp`)
- CLI orchestration → CLI output: prints generation (evidence: `main.cpp`)
- User → HTTP client: sends request (evidence: `README.md`)
- HTTP client → HTTP server: connects (evidence: `README.md`)
- CLI orchestration → HTTP server: starts service (evidence: `main.cpp`)
- HTTP server → HTTP API routes: dispatches request (evidence: `src/server/http_server.cpp`)
- HTTP API routes → Tokenizer and templates: encodes prompt (evidence: `src/server/http_server.cpp`)
- HTTP API routes → Transformer engine: forwards tokens (evidence: `src/server/http_server.cpp`)
- HTTP API routes → Token sampler: samples logits (evidence: `src/server/http_server.cpp`)
- HTTP API routes → Tokenizer and templates: decodes tokens (evidence: `src/server/http_server.cpp`)
- HTTP API routes → JSON / SSE response: formats output (evidence: `src/server/http_server.cpp`)
- JSON / SSE response → HTTP client: returns response (evidence: `README.md`)
