How binbinao/cllm_inference works
cllm_inference is a C++20 CPU GGUF inference engine with two user-facing modes: local CLI generation and an OpenAI-compatible HTTP/SSE service. Both load a model, tokenize a prompt, run autoregressive Transformer inference, sample tokens, and decode output. The diagram separates model parsing, inference kernels, and serving; the README documents additional GGUF, quantization, tokenizer, and API capabilities beyond the sampled excerpts. Edges are limited to excerpted calls or explicit README relationships; several implementation details were not sampled.
Main entry points: main.cpp CLI orchestration · http_server.cpp HTTP API routes · engine.cpp Transformer engine · tokenizer.cpp Tokenizer and templates
Components 15
Entry points
- CLI orchestration
main.cpp - CLI output
main.cpp
Model and tokens
- GGUF loader
src/core/gguf.cpp - Mapped tensors
include/cllm/core/tensor.hpp - Weight dequantization
src/core/quant.cpp - Tokenizer and templates
src/core/tokenizer.cpp
Inference runtime
- Transformer engine
src/core/engine.cpp - Token sampler
src/core/sampler.cpp - Thread pool
src/core/thread_pool.cpp
HTTP serving
- HTTP server
src/server/http_server.cpp - HTTP API routes
src/server/http_server.cpp - JSON / SSE response
src/server/http_server.cpp
Other components
- User
- GGUF model file
- HTTP client
Connections 25
- User → CLI orchestration: provides options
README.md - CLI orchestration → GGUF loader: loads model
main.cpp - GGUF model file → GGUF loader: reads GGUF
README.md - GGUF loader → Mapped tensors: maps tensors
README.md - Mapped tensors → Weight dequantization: dequantizes with
include/cllm/core/tensor.hpp - CLI orchestration → Transformer engine: initializes engine
main.cpp - CLI orchestration → Tokenizer and templates: initializes tokenizer
main.cpp - Transformer engine → Mapped tensors: dequantizes weights
src/core/engine.cpp - Transformer engine → Thread pool: parallelizes matmul
src/core/engine.cpp - CLI orchestration → Thread pool: creates pool
main.cpp - CLI orchestration → Tokenizer and templates: encodes prompt
main.cpp - CLI orchestration → Transformer engine: forwards tokens
main.cpp - CLI orchestration → Token sampler: samples logits
main.cpp - CLI orchestration → Tokenizer and templates: decodes tokens
main.cpp - CLI orchestration → CLI output: prints generation
main.cpp - User → HTTP client: sends request
README.md - HTTP client → HTTP server: connects
README.md - CLI orchestration → HTTP server: starts service
main.cpp - HTTP server → HTTP API routes: dispatches request
src/server/http_server.cpp - HTTP API routes → Tokenizer and templates: encodes prompt
src/server/http_server.cpp - HTTP API routes → Transformer engine: forwards tokens
src/server/http_server.cpp - HTTP API routes → Token sampler: samples logits
src/server/http_server.cpp - HTTP API routes → Tokenizer and templates: decodes tokens
src/server/http_server.cpp - HTTP API routes → JSON / SSE response: formats output
src/server/http_server.cpp - JSON / SSE response → HTTP client: returns response
README.md