# microsoft/llama.cpp-nd-usb4 architecture

> Architecture diagram of the GitHub repository microsoft/llama.cpp-nd-usb4, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/llama.cpp-nd-usb4
- Repository: https://github.com/microsoft/llama.cpp-nd-usb4

## Overview

llama.cpp is a C/C++ inference runtime for local and cloud LLM and multimodal models, exposed through a library, command-line tools, and an HTTP server. The principal workflow is to obtain a model, load it through the runtime, execute inference on ggml backends, and return generated output to a client. The graph also preserves model conversion, multimodal processing, and optional distributed RPC capabilities. Edges are limited to README statements and sampled/indexed source references; several unsampled implementation connections are intentionally omitted.

Main entry points: [`src/llama-context.cpp`](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/src/llama-context.cpp) (Inference Context), [`ggml/include/ggml-backend.h`](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/ggml/include/ggml-backend.h) (Backend Scheduling), [`tools/server/server.cpp`](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/tools/server/server.cpp) (HTTP Server), [`common/download.cpp`](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/common/download.cpp) (Model Download)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_model["Model Preparation"]
  node_download["Model Download<br/>[download.cpp]"]
  node_convert["HF Conversion"]
  node_gguf["GGUF Model"]
end

subgraph group_runtime["Inference Runtime"]
  node_api["llama API<br/>[llama.h]"]
  node_context["Inference Context<br/>[llama-context.cpp]"]
  node_sampling["Sampling<br/>[sampling.cpp]"]
  node_grammar["Grammar Parsing<br/>[peg-parser.cpp]"]
end

subgraph group_compute["Compute Backends"]
  node_ggml["Tensor Operations<br/>[ggml.h]"]
  node_backend["Backend Scheduling<br/>[ggml-backend.h]"]
  node_cpu["CPU Backend<br/>[ggml-cpu.cpp]"]
  node_accelerators["Accelerator Backends<br/>[ggml-metal.cpp]"]
end

subgraph group_interfaces["User Interfaces"]
  node_cli["CLI<br/>[cli-client.cpp]"]
  node_server["HTTP Server<br/>[server.cpp]"]
  node_chat["Chat Application<br/>[server.svelte.ts]"]
end

subgraph group_extensions["Multimodal and RPC"]
  node_multimodal["Multimodal Processing"]
  node_tts["Speech Generation<br/>[tts.cpp]"]
  node_rpc["RPC Server<br/>[rpc-server.cpp]"]
end

node_user(("User"))
node_hf["Hugging Face"]
node_remote(("Remote Hosts"))

node_user -->|"selects model"| node_hf
node_hf -->|"provides weights"| node_download
node_download -.->|"stores model"| node_gguf
node_user -.->|"converts model"| node_convert
node_convert -.->|"produces GGUF"| node_gguf
node_user -->|"runs inference"| node_cli
node_user -->|"requests inference"| node_server
node_chat -->|"uses server"| node_server
node_cli -.->|"invokes runtime"| node_context
node_server -.->|"invokes runtime"| node_context
node_server -->|"routes API calls"| node_api
node_context -->|"builds computations"| node_ggml
node_context -->|"schedules execution"| node_backend
node_backend -.->|"dispatches work"| node_cpu
node_backend -.->|"dispatches work"| node_accelerators
node_context -.->|"selects tokens"| node_sampling
node_context -.->|"constrains output"| node_grammar
node_context -.->|"supports multimodal"| node_multimodal
node_user -.->|"requests speech"| node_tts
node_remote -->|"hosts compute"| node_rpc
node_rpc -.->|"serves execution"| node_backend

click node_download "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/common/download.cpp"
click node_convert "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/convert_hf_to_gguf.py"
click node_api "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/include/llama.h"
click node_context "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/src/llama-context.cpp"
click node_sampling "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/common/sampling.cpp"
click node_grammar "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/common/peg-parser.cpp"
click node_ggml "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/ggml/include/ggml.h"
click node_backend "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/ggml/include/ggml-backend.h"
click node_cpu "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/ggml/src/ggml-cpu/ggml-cpu.cpp"
click node_accelerators "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/ggml/src/ggml-metal/ggml-metal.cpp"
click node_cli "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/tools/cli/cli-client.cpp"
click node_server "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/tools/server/server.cpp"
click node_chat "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/ui/apps/chat/src/lib/stores/server.svelte.ts"
click node_multimodal "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/tools/mtmd/mtmd-subcontext.cpp"
click node_tts "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/tools/tts/tts.cpp"
click node_rpc "https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/tools/rpc/rpc-server.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_download,node_convert,node_gguf,node_user toneBlue
class node_api,node_context,node_sampling,node_grammar toneAmber
class node_ggml,node_backend,node_cpu,node_accelerators toneMint
class node_cli,node_server,node_chat toneRose
class node_multimodal,node_tts,node_rpc,node_hf,node_remote toneIndigo
```

## Components

### Model Preparation

- [Model Download](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/common/download.cpp): `common/download.cpp`
- [HF Conversion](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/convert_hf_to_gguf.py): `convert_hf_to_gguf.py`
- **GGUF Model**

### Inference Runtime

- [llama API](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/include/llama.h): `include/llama.h`
- [Inference Context](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/src/llama-context.cpp): `src/llama-context.cpp`
- [Sampling](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/common/sampling.cpp): `common/sampling.cpp`
- [Grammar Parsing](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/common/peg-parser.cpp): `common/peg-parser.cpp`

### Compute Backends

- [Tensor Operations](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/ggml/include/ggml.h): `ggml/include/ggml.h`
- [Backend Scheduling](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/ggml/include/ggml-backend.h): `ggml/include/ggml-backend.h`
- [CPU Backend](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/ggml/src/ggml-cpu/ggml-cpu.cpp): `ggml/src/ggml-cpu/ggml-cpu.cpp`
- [Accelerator Backends](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/ggml/src/ggml-metal/ggml-metal.cpp): `ggml/src/ggml-metal/ggml-metal.cpp`

### User Interfaces

- [CLI](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/tools/cli/cli-client.cpp): `tools/cli/cli-client.cpp`
- [HTTP Server](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/tools/server/server.cpp): `tools/server/server.cpp`
- [Chat Application](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/ui/apps/chat/src/lib/stores/server.svelte.ts): `ui/apps/chat/src/lib/stores/server.svelte.ts`

### Multimodal and RPC

- [Multimodal Processing](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/tools/mtmd/mtmd-subcontext.cpp): `tools/mtmd/mtmd-subcontext.cpp`
- [Speech Generation](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/tools/tts/tts.cpp): `tools/tts/tts.cpp`
- [RPC Server](https://github.com/microsoft/llama.cpp-nd-usb4/blob/master/tools/rpc/rpc-server.cpp): `tools/rpc/rpc-server.cpp`

### Other components

- **User**
- **Hugging Face**
- **Remote Hosts**

## Connections

- User → Hugging Face: selects model (evidence: `README.md`)
- Hugging Face → Model Download: provides weights (evidence: `README.md`)
- Model Download → GGUF Model: stores model
- User → HF Conversion: converts model
- HF Conversion → GGUF Model: produces GGUF
- User → CLI: runs inference (evidence: `README.md`)
- User → HTTP Server: requests inference (evidence: `README.md`)
- Chat Application → HTTP Server: uses server (evidence: `README.md`)
- CLI → Inference Context: invokes runtime
- HTTP Server → Inference Context: invokes runtime
- HTTP Server → llama API: routes API calls (evidence: `tools/server/server.cpp`)
- Inference Context → Tensor Operations: builds computations (evidence: `src/llama-context.cpp`)
- Inference Context → Backend Scheduling: schedules execution (evidence: `src/llama-context.cpp`)
- Backend Scheduling → CPU Backend: dispatches work
- Backend Scheduling → Accelerator Backends: dispatches work
- Inference Context → Sampling: selects tokens
- Inference Context → Grammar Parsing: constrains output
- Inference Context → Multimodal Processing: supports multimodal
- User → Speech Generation: requests speech
- Remote Hosts → RPC Server: hosts compute (evidence: `README.md`)
- RPC Server → Backend Scheduling: serves execution
