# arm/ai-chat architecture

> Architecture diagram of the GitHub repository arm/ai-chat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arm/ai-chat
- Repository: https://github.com/arm/ai-chat

## Overview

llama.cpp is a local and cloud LLM inference runtime, exposed through a C/C++ API, command-line tools, and an OpenAI-compatible server. A typical user supplies a GGUF model and prompt; the runtime loads the model, builds and executes inference graphs on available backends, and returns generated output. The README also documents model conversion and quantization, multimodal server support, and a browser UI. Internal links are limited to relationships shown by excerpts or the source index; unsampled runtime wiring is not inferred.

Main entry points: [`tools/main/main.cpp`](https://github.com/arm/ai-chat/blob/master/tools/main/main.cpp) (CLI inference), [`tools/server/server.cpp`](https://github.com/arm/ai-chat/blob/master/tools/server/server.cpp) (HTTP server), [`src/llama-context.cpp`](https://github.com/arm/ai-chat/blob/master/src/llama-context.cpp) (Inference context), [`ggml/src/ggml-backend.cpp`](https://github.com/arm/ai-chat/blob/master/ggml/src/ggml-backend.cpp) (Backend scheduler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User interfaces"]
  node_cli["CLI inference<br/>[main.cpp]"]
  node_server["HTTP server<br/>[server.cpp]"]
  node_webui["Browser UI<br/>[ChatScreen.svelte]"]
  node_api["C API<br/>[llama.h]"]
end

subgraph group_runtime["Inference runtime"]
  node_common["Runtime options<br/>[common.h]"]
  node_chat["Chat formatting<br/>[chat.h]"]
  node_chatparse["Tool-call parsing"]
  node_context["Inference context<br/>[llama-context.cpp]"]
  node_model["Model loading<br/>[llama-model.cpp]"]
  node_sampling["Token sampling<br/>[llama-sampling.cpp]"]
  node_memory[("KV and memory<br/>[llama-memory.cpp]")]
  node_modelarch["Model architectures"]
end

subgraph group_compute["Tensor compute"]
  node_ggml["Tensor operations<br/>[ggml.cpp]"]
  node_backend["Backend scheduler<br/>[ggml-backend.cpp]"]
  node_cpu["CPU kernels<br/>[ops.cpp]"]
  node_gpu["Accelerator backends"]
end

subgraph group_models["Model assets"]
  node_gguf["GGUF reader/writer<br/>[gguf.py]"]
  node_convert["Model conversion"]
  node_quantize["Model quantization<br/>[quantize.cpp]"]
end

subgraph group_multimodal["Multimodal inputs"]
  node_mtmd["Multimodal processing<br/>[mtmd.cpp]"]
  node_clip["Image encoder<br/>[clip.cpp]"]
  node_audio["Audio processing<br/>[mtmd-audio.cpp]"]
end

node_user(("User"))
node_hf(("Hugging Face"))

node_user -->|"runs prompt"| node_cli
node_user -->|"sends requests"| node_server
node_user -.->|"chats through"| node_webui
node_cli -->|"parses options"| node_common
node_cli -->|"initializes runtime"| node_context
node_cli -->|"formats messages"| node_chat
node_cli -->|"uses API"| node_api
node_server -->|"uses runtime options"| node_common
node_server -->|"uses chat support"| node_chat
node_server -->|"uses API"| node_api
node_server -->|"supports multimodal"| node_mtmd
node_context -->|"uses model"| node_model
node_context -->|"uses memory"| node_memory
node_context -->|"schedules graph"| node_backend
node_context -.->|"builds compute graph"| node_ggml
node_modelarch -->|"defines architectures"| node_model
node_modelarch -->|"uses recurrent memory"| node_memory
node_backend -.->|"dispatches kernels"| node_cpu
node_backend -.->|"dispatches kernels"| node_gpu
node_convert -.->|"writes model format"| node_gguf
node_quantize -.->|"produces quantized model"| node_gguf
node_hf -->|"downloads model"| node_cli
node_mtmd -->|"uses image encoder"| node_clip
node_mtmd -.->|"handles audio"| node_audio

click node_cli "https://github.com/arm/ai-chat/blob/master/tools/main/main.cpp"
click node_server "https://github.com/arm/ai-chat/blob/master/tools/server/server.cpp"
click node_webui "https://github.com/arm/ai-chat/blob/master/tools/server/webui/src/lib/components/app/chat/ChatScreen/ChatScreen.svelte"
click node_api "https://github.com/arm/ai-chat/blob/master/include/llama.h"
click node_common "https://github.com/arm/ai-chat/blob/master/common/common.h"
click node_chat "https://github.com/arm/ai-chat/blob/master/common/chat.h"
click node_chatparse "https://github.com/arm/ai-chat/blob/master/common/chat-parser-xml-toolcall.cpp"
click node_context "https://github.com/arm/ai-chat/blob/master/src/llama-context.cpp"
click node_model "https://github.com/arm/ai-chat/blob/master/src/llama-model.cpp"
click node_sampling "https://github.com/arm/ai-chat/blob/master/src/llama-sampling.cpp"
click node_memory "https://github.com/arm/ai-chat/blob/master/src/llama-memory.cpp"
click node_modelarch "https://github.com/arm/ai-chat/tree/master/src/models"
click node_ggml "https://github.com/arm/ai-chat/blob/master/ggml/src/ggml.cpp"
click node_backend "https://github.com/arm/ai-chat/blob/master/ggml/src/ggml-backend.cpp"
click node_cpu "https://github.com/arm/ai-chat/blob/master/ggml/src/ggml-cpu/ops.cpp"
click node_gpu "https://github.com/arm/ai-chat/tree/master/ggml/src/ggml-cuda"
click node_gguf "https://github.com/arm/ai-chat/blob/master/gguf-py/gguf/gguf.py"
click node_convert "https://github.com/arm/ai-chat/blob/master/convert_hf_to_gguf.py"
click node_quantize "https://github.com/arm/ai-chat/blob/master/tools/quantize/quantize.cpp"
click node_mtmd "https://github.com/arm/ai-chat/blob/master/tools/mtmd/mtmd.cpp"
click node_clip "https://github.com/arm/ai-chat/blob/master/tools/mtmd/clip.cpp"
click node_audio "https://github.com/arm/ai-chat/blob/master/tools/mtmd/mtmd-audio.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_server,node_webui,node_api,node_user toneBlue
class node_common,node_chat,node_chatparse,node_context,node_model,node_sampling,node_memory,node_modelarch toneAmber
class node_ggml,node_backend,node_cpu,node_gpu toneMint
class node_gguf,node_convert,node_quantize toneRose
class node_mtmd,node_clip,node_audio,node_hf toneIndigo
```

## Components

### User interfaces

- [CLI inference](https://github.com/arm/ai-chat/blob/master/tools/main/main.cpp): `tools/main/main.cpp`
- [HTTP server](https://github.com/arm/ai-chat/blob/master/tools/server/server.cpp): `tools/server/server.cpp`
- [Browser UI](https://github.com/arm/ai-chat/blob/master/tools/server/webui/src/lib/components/app/chat/ChatScreen/ChatScreen.svelte): `tools/server/webui/src/lib/components/app/chat/ChatScreen/ChatScreen.svelte`
- [C API](https://github.com/arm/ai-chat/blob/master/include/llama.h): `include/llama.h`

### Inference runtime

- [Runtime options](https://github.com/arm/ai-chat/blob/master/common/common.h): `common/common.h`
- [Chat formatting](https://github.com/arm/ai-chat/blob/master/common/chat.h): `common/chat.h`
- [Tool-call parsing](https://github.com/arm/ai-chat/blob/master/common/chat-parser-xml-toolcall.cpp): `common/chat-parser-xml-toolcall.cpp`
- [Inference context](https://github.com/arm/ai-chat/blob/master/src/llama-context.cpp): `src/llama-context.cpp`
- [Model loading](https://github.com/arm/ai-chat/blob/master/src/llama-model.cpp): `src/llama-model.cpp`
- [Token sampling](https://github.com/arm/ai-chat/blob/master/src/llama-sampling.cpp): `src/llama-sampling.cpp`
- [KV and memory](https://github.com/arm/ai-chat/blob/master/src/llama-memory.cpp): `src/llama-memory.cpp`
- [Model architectures](https://github.com/arm/ai-chat/tree/master/src/models): `src/models`

### Tensor compute

- [Tensor operations](https://github.com/arm/ai-chat/blob/master/ggml/src/ggml.cpp): `ggml/src/ggml.cpp`
- [Backend scheduler](https://github.com/arm/ai-chat/blob/master/ggml/src/ggml-backend.cpp): `ggml/src/ggml-backend.cpp`
- [CPU kernels](https://github.com/arm/ai-chat/blob/master/ggml/src/ggml-cpu/ops.cpp): `ggml/src/ggml-cpu/ops.cpp`
- [Accelerator backends](https://github.com/arm/ai-chat/tree/master/ggml/src/ggml-cuda): `ggml/src/ggml-cuda`

### Model assets

- [GGUF reader/writer](https://github.com/arm/ai-chat/blob/master/gguf-py/gguf/gguf.py): `gguf-py/gguf/gguf.py`
- [Model conversion](https://github.com/arm/ai-chat/blob/master/convert_hf_to_gguf.py): `convert_hf_to_gguf.py`
- [Model quantization](https://github.com/arm/ai-chat/blob/master/tools/quantize/quantize.cpp): `tools/quantize/quantize.cpp`

### Multimodal inputs

- [Multimodal processing](https://github.com/arm/ai-chat/blob/master/tools/mtmd/mtmd.cpp): `tools/mtmd/mtmd.cpp`
- [Image encoder](https://github.com/arm/ai-chat/blob/master/tools/mtmd/clip.cpp): `tools/mtmd/clip.cpp`
- [Audio processing](https://github.com/arm/ai-chat/blob/master/tools/mtmd/mtmd-audio.cpp): `tools/mtmd/mtmd-audio.cpp`

### Other components

- **User**
- **Hugging Face**

## Connections

- User → CLI inference: runs prompt
- User → HTTP server: sends requests
- User → Browser UI: chats through
- CLI inference → Runtime options: parses options (evidence: `tools/main/main.cpp`)
- CLI inference → Inference context: initializes runtime (evidence: `tools/main/main.cpp`)
- CLI inference → Chat formatting: formats messages (evidence: `tools/main/main.cpp`)
- CLI inference → C API: uses API (evidence: `tools/main/main.cpp`)
- HTTP server → Runtime options: uses runtime options (evidence: `tools/server/server.cpp`)
- HTTP server → Chat formatting: uses chat support (evidence: `tools/server/server.cpp`)
- HTTP server → C API: uses API (evidence: `tools/server/server.cpp`)
- HTTP server → Multimodal processing: supports multimodal (evidence: `tools/server/server.cpp`)
- Inference context → Model loading: uses model (evidence: `src/llama-context.cpp`)
- Inference context → KV and memory: uses memory (evidence: `src/llama-context.cpp`)
- Inference context → Backend scheduler: schedules graph (evidence: `src/llama-context.cpp`)
- Inference context → Tensor operations: builds compute graph
- Model architectures → Model loading: defines architectures (evidence: `src/models/models.h`)
- Model architectures → KV and memory: uses recurrent memory (evidence: `src/models/models.h`)
- Backend scheduler → CPU kernels: dispatches kernels
- Backend scheduler → Accelerator backends: dispatches kernels
- Model conversion → GGUF reader/writer: writes model format
- Model quantization → GGUF reader/writer: produces quantized model
- Hugging Face → CLI inference: downloads model
- Multimodal processing → Image encoder: uses image encoder (evidence: `tools/mtmd/clip.cpp`)
- Multimodal processing → Audio processing: handles audio