arm/ai-chat

How arm/ai-chat works

llama.cpp is a local and cloud LLM inference runtime, exposed through a C/C++ API, command-line tools, and an OpenAI-compatible server. A typical user supplies a GGUF model and prompt; the runtime loads the model, builds and executes inference graphs on available backends, and returns generated output. The README also documents model conversion and quantization, multimodal server support, and a browser UI. Internal links are limited to relationships shown by excerpts or the source index; unsampled runtime wiring is not inferred.

Main entry points: main.cpp CLI inference · server.cpp HTTP server · llama-context.cpp Inference context · ggml-backend.cpp Backend scheduler

Components 24

User interfaces

CLI inference tools/main/main.cpp
HTTP server tools/server/server.cpp
Browser UI tools/server/webui/src/lib/components/app/chat/ChatScreen/ChatScreen.svelte
C API include/llama.h

Inference runtime

Runtime options common/common.h
Chat formatting common/chat.h
Tool-call parsing common/chat-parser-xml-toolcall.cpp
Inference context src/llama-context.cpp
Model loading src/llama-model.cpp
Token sampling src/llama-sampling.cpp
KV and memory src/llama-memory.cpp
Model architectures src/models

Tensor compute

Tensor operations ggml/src/ggml.cpp
Backend scheduler ggml/src/ggml-backend.cpp
CPU kernels ggml/src/ggml-cpu/ops.cpp
Accelerator backends ggml/src/ggml-cuda

Model assets

GGUF reader/writer gguf-py/gguf/gguf.py
Model conversion convert_hf_to_gguf.py
Model quantization tools/quantize/quantize.cpp

Multimodal inputs

Multimodal processing tools/mtmd/mtmd.cpp
Image encoder tools/mtmd/clip.cpp
Audio processing tools/mtmd/mtmd-audio.cpp

Other components

User
Hugging Face

Connections 24

User → CLI inference: runs prompt
User → HTTP server: sends requests
User → Browser UI: chats through
CLI inference → Runtime options: parses options tools/main/main.cpp
CLI inference → Inference context: initializes runtime tools/main/main.cpp
CLI inference → Chat formatting: formats messages tools/main/main.cpp
CLI inference → C API: uses API tools/main/main.cpp
HTTP server → Runtime options: uses runtime options tools/server/server.cpp
HTTP server → Chat formatting: uses chat support tools/server/server.cpp
HTTP server → C API: uses API tools/server/server.cpp
HTTP server → Multimodal processing: supports multimodal tools/server/server.cpp
Inference context → Model loading: uses model src/llama-context.cpp
Inference context → KV and memory: uses memory src/llama-context.cpp
Inference context → Backend scheduler: schedules graph src/llama-context.cpp
Inference context → Tensor operations: builds compute graph
Model architectures → Model loading: defines architectures src/models/models.h
Model architectures → KV and memory: uses recurrent memory src/models/models.h
Backend scheduler → CPU kernels: dispatches kernels
Backend scheduler → Accelerator backends: dispatches kernels
Model conversion → GGUF reader/writer: writes model format
Model quantization → GGUF reader/writer: produces quantized model
Hugging Face → CLI inference: downloads model
Multimodal processing → Image encoder: uses image encoder tools/mtmd/clip.cpp
Multimodal processing → Audio processing: handles audio

More C++ diagrams: zkevz/voxel-engine · aidanshipperley/ppo-libtorch · amkarahalios/dd_vrptw · articysoftware/articyximporterforunreal · cbachhuber/g2gdelay

Drawn by GitDiagram on October 1, 2026 from the repository's file tree, README and source files. Read as Markdown ·