# withcatai/node-llama-cpp architecture

> Architecture diagram of the GitHub repository withcatai/node-llama-cpp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/withcatai/node-llama-cpp
- Repository: https://github.com/withcatai/node-llama-cpp

## Overview

#### Purpose and entry points
`node-llama-cpp` is a TypeScript/Node.js SDK and CLI for local GGUF LLM inference through native `llama.cpp` bindings. The public library barrel is `src/index.ts`; runtime initialization begins with `src/bindings/getLlama.ts`. The terminal interface starts at `src/cli/cli.ts` and dispatches commands such as chat, completion, model download, inspection, and source builds.

#### Native binding and runtime selection
The core boundary is TypeScript-to-C++ Node addon. `llama/addon/addon.cpp` registers native exports; object wrappers such as `llama/addon/AddonModel.cpp` and `llama/addon/AddonGrammar.cpp` own native model and constraint state. TypeScript binding facades in `src/bindings/Llama.ts` and `src/bindings/AddonTypes.ts` expose this functionality safely to the evaluator layer.
At startup, binding resolution selects compatible prebuilt platform packages or falls back to obtaining and compiling `llama.cpp`. This path is coordinated by `src/bindings/getLlama.ts`, `src/bindings/utils/resolveActualBindingBinaryPath.ts`, and `src/bindings/utils/compileLLamaCpp.ts`.

#### Hardware, builds, and binary distribution
Compute backend selection detects available CPU/GPU capabilities and chooses Metal, CUDA, or Vulkan variants where applicable. Key decision points are `src/bindings/utils/detectAvailableComputeLayers.ts`, `src/bindings/utils/getBestComputeLayersAvailable.ts`, and native Vulkan probing in `llama/gpuInfo/vulkan-gpu-info.cpp`.
CMake drives native compilation through `llama/CMakeLists.txt`, with cross-platform toolchains under `llama/toolchains/`. Standalone NPM packages distribute platform/backend-specific binaries, e.g. `packages/@node-llama-cpp/linux-x64-cuda/src/index.ts` and `packages/@node-llama-cpp/mac-arm64-metal/src/index.ts`.

#### Model lifecycle and inference
Application flow is: initialize `Llama` → load a GGUF model → create a context/sequences → tokenize/evaluate/sample generated tokens → dispose native resources. Main abstractions are `src/evaluator/LlamaModel/LlamaModel.ts`, `src/evaluator/LlamaContext/LlamaContext.ts`, and `src/evaluator/LlamaContext/LlamaSampler.ts`. The context layer manages batching, parallel work prioritization, checkpoints, token prediction, and memory-sensitive inference boundaries.

#### Chat orchestration and structured generation
`LlamaChatSession` maintains history and turns prompts into context evaluation, while `LlamaChat` supplies function-calling and response orchestration. Primary paths: `src/evaluator/LlamaChatSession/LlamaChatSession.ts`, `src/evaluator/LlamaChat/LlamaChat.ts`, and `src/ChatWrapper.ts`.
Model-family chat formatting is isolated behind specialized and template-driven wrappers, resolved by `src/chatWrappers/utils/resolveChatWrapper.ts`; generic templates are handled in `src/chatWrappers/generic/JinjaTemplateChatWrapper.ts`. Grammar classes constrain decoding for JSON/schema and function-call output: `src/evaluator/LlamaGrammar.ts` and `src/evaluator/LlamaJsonSchemaGrammar.ts`.

#### GGUF I/O and model acquisition
The GGUF subsystem parses local or fetched model metadata, tensors, split files, and resource estimates independently of model loading. Its boundary begins at `src/gguf/readGgufFileInfo.ts`, with version parsers in `src/gguf/parser/parseGguf.ts` and readers including `src/gguf/fileReaders/GgufNetworkFetchFileReader.ts`.
CLI/model utilities resolve model URIs and download destinations, including GitHub/Hugging Face-style endpoints, via `src/utils/createModelDownloader.ts`, `src/utils/parseModelUri.ts`, and `src/utils/modelDownloadEndpoints.ts`.

#### CLI, scaffolding, and documentation
CLI commands are an alternative integration surface over the same library lifecycle: `src/cli/commands/ChatCommand.ts`, `src/cli/commands/PullCommand.ts`, and `src/cli/commands/source/SourceCommand.ts`. Project generation is separately packaged in `packages/create-node-llama-cpp/src/cli.ts`; templates include Node and Electron applications. VitePress documentation is built from `docs/` and configured at `.vitepress/config.ts`.

Main entry points: [`src/bindings/utils/resolveActualBindingBinaryPath.ts`](https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/utils/resolveActualBindingBinaryPath.ts) (Binary resolution), [`src/bindings/getLlama.ts`](https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/getLlama.ts) (Runtime initialization), [`src/cli/cli.ts`](https://github.com/withcatai/node-llama-cpp/blob/master/src/cli/cli.ts) (CLI dispatcher), [`src/bindings/Llama.ts`](https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/Llama.ts) (Binding facade)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_public["Public surfaces"]
  node_library["Library barrel<br/>TypeScript API<br/>[index.ts]"]
  node_cli["CLI dispatcher<br/>Node CLI<br/>[cli.ts]"]
end

subgraph group_runtime["Runtime and native binding"]
  node_get_llama["Runtime initialization<br/>binding loader<br/>[getLlama.ts]"]
  node_binding_facade["Binding facade<br/>TypeScript binding API<br/>[Llama.ts]"]
  node_binding_resolution["Binary resolution<br/>platform resolver"]
  node_compute_selection["Compute backend selection<br/>hardware detection"]
  node_native_build["Native compilation<br/>llama.cpp build<br/>[compileLLamaCpp.ts]"]
  node_native_addon{{"Node native addon<br/>C++ N-API addon<br/>[addon.cpp]"}}
  node_cmake["CMake build<br/>native build definition<br/>[CMakeLists.txt]"]
end

subgraph group_inference["Inference and chat"]
  node_model["Llama model<br/>model lifecycle<br/>[LlamaModel.ts]"]
  node_context["Llama context<br/>inference runtime<br/>[LlamaContext.ts]"]
  node_sampler["Token sampler<br/>decoding<br/>[LlamaSampler.ts]"]
  node_chat_session["Chat session<br/>conversation state"]
  node_chat_orchestration["Chat orchestration<br/>function calling<br/>[LlamaChat.ts]"]
  node_chat_wrappers["Chat wrapper resolver<br/>prompt formatting"]
  node_grammar["Structured grammar<br/>decoding constraints"]
end

subgraph group_models["Models and GGUF"]
  node_gguf_reader["GGUF metadata reader<br/>GGUF I/O"]
  node_gguf_parser["GGUF parser<br/>format parser<br/>[parseGguf.ts]"]
  node_model_downloader["Model downloader<br/>model acquisition"]
end

subgraph group_delivery["Distribution and tooling"]
  node_binary_packages["Platform binary packages<br/>NPM binary distribution<br/>[index.ts]"]
  node_create_cli["Project generator<br/>scaffolding CLI<br/>[cli.ts]"]
  node_docs["Documentation site<br/>VitePress<br/>[config.ts]"]
end

node_library -->|"initializes"| node_get_llama
node_cli -->|"initializes"| node_get_llama
node_cli -->|"chat command"| node_chat_session
node_cli -->|"pull command"| node_model_downloader
node_get_llama -->|"resolves binary"| node_binding_resolution
node_binding_resolution -->|"matches backend"| node_compute_selection
node_binding_resolution -->|"loads prebuilt"| node_binary_packages
node_binding_resolution -.->|"fallback build"| node_native_build
node_native_build -->|"invokes"| node_cmake
node_cmake -->|"builds"| node_native_addon
node_binary_packages -.->|"distribute"| node_native_addon
node_get_llama -->|"returns"| node_binding_facade
node_binding_facade -->|"calls"| node_native_addon
node_binding_facade -->|"creates"| node_model
node_model -->|"loads model data"| node_gguf_reader
node_gguf_reader -->|"parses"| node_gguf_parser
node_model_downloader -->|"provides model file"| node_gguf_reader
node_model -->|"creates"| node_context
node_context -->|"samples tokens"| node_sampler
node_chat_session -->|"orchestrates turns"| node_chat_orchestration
node_chat_session -->|"evaluates prompts"| node_context
node_chat_orchestration -->|"formats prompts"| node_chat_wrappers
node_chat_orchestration -->|"constrains output"| node_grammar
node_grammar -->|"guides decoding"| node_sampler
node_create_cli -.->|"scaffolds usage"| node_library
node_docs -.->|"documents"| node_library

click node_library "https://github.com/withcatai/node-llama-cpp/blob/master/src/index.ts"
click node_cli "https://github.com/withcatai/node-llama-cpp/blob/master/src/cli/cli.ts"
click node_get_llama "https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/getLlama.ts"
click node_binding_facade "https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/Llama.ts"
click node_binding_resolution "https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/utils/resolveActualBindingBinaryPath.ts"
click node_compute_selection "https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/utils/getBestComputeLayersAvailable.ts"
click node_native_build "https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/utils/compileLLamaCpp.ts"
click node_native_addon "https://github.com/withcatai/node-llama-cpp/blob/master/llama/addon/addon.cpp"
click node_cmake "https://github.com/withcatai/node-llama-cpp/blob/master/llama/CMakeLists.txt"
click node_model "https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaModel/LlamaModel.ts"
click node_context "https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaContext/LlamaContext.ts"
click node_sampler "https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaContext/LlamaSampler.ts"
click node_chat_session "https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaChatSession/LlamaChatSession.ts"
click node_chat_orchestration "https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaChat/LlamaChat.ts"
click node_chat_wrappers "https://github.com/withcatai/node-llama-cpp/blob/master/src/chatWrappers/utils/resolveChatWrapper.ts"
click node_grammar "https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaJsonSchemaGrammar.ts"
click node_gguf_reader "https://github.com/withcatai/node-llama-cpp/blob/master/src/gguf/readGgufFileInfo.ts"
click node_gguf_parser "https://github.com/withcatai/node-llama-cpp/blob/master/src/gguf/parser/parseGguf.ts"
click node_model_downloader "https://github.com/withcatai/node-llama-cpp/blob/master/src/utils/createModelDownloader.ts"
click node_binary_packages "https://github.com/withcatai/node-llama-cpp/blob/master/packages/%40node-llama-cpp/linux-x64-cuda/src/index.ts"
click node_create_cli "https://github.com/withcatai/node-llama-cpp/blob/master/packages/create-node-llama-cpp/src/cli.ts"
click node_docs "https://github.com/withcatai/node-llama-cpp/blob/master/.vitepress/config.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_library,node_cli toneBlue
class node_get_llama,node_binding_facade,node_binding_resolution,node_compute_selection,node_native_build,node_native_addon,node_cmake toneAmber
class node_model,node_context,node_sampler,node_chat_session,node_chat_orchestration,node_chat_wrappers,node_grammar toneMint
class node_gguf_reader,node_gguf_parser,node_model_downloader toneRose
class node_binary_packages,node_create_cli,node_docs toneIndigo
```

## Components

### Public surfaces

- [Library barrel](https://github.com/withcatai/node-llama-cpp/blob/master/src/index.ts): TypeScript API · Public SDK exports. · `src/index.ts`
- [CLI dispatcher](https://github.com/withcatai/node-llama-cpp/blob/master/src/cli/cli.ts): Node CLI · `src/cli/cli.ts`

### Runtime and native binding

- [Runtime initialization](https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/getLlama.ts): binding loader · Resolves and initializes a compatible native binding. · `src/bindings/getLlama.ts`
- [Binding facade](https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/Llama.ts): TypeScript binding API · `src/bindings/Llama.ts`
- [Binary resolution](https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/utils/resolveActualBindingBinaryPath.ts): platform resolver · Selects a prebuilt binary or native build output. · `src/bindings/utils/resolveActualBindingBinaryPath.ts`
- [Compute backend selection](https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/utils/getBestComputeLayersAvailable.ts): hardware detection · Chooses CPU, Metal, CUDA, or Vulkan capability. · `src/bindings/utils/getBestComputeLayersAvailable.ts`
- [Native compilation](https://github.com/withcatai/node-llama-cpp/blob/master/src/bindings/utils/compileLLamaCpp.ts): llama.cpp build · `src/bindings/utils/compileLLamaCpp.ts`
- [Node native addon](https://github.com/withcatai/node-llama-cpp/blob/master/llama/addon/addon.cpp): C++ N-API addon · Registers native exports and resource wrappers. · `llama/addon/addon.cpp`
- [CMake build](https://github.com/withcatai/node-llama-cpp/blob/master/llama/CMakeLists.txt): native build definition · `llama/CMakeLists.txt`

### Inference and chat

- [Llama model](https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaModel/LlamaModel.ts): model lifecycle · Loads GGUF models and owns native model state. · `src/evaluator/LlamaModel/LlamaModel.ts`
- [Llama context](https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaContext/LlamaContext.ts): inference runtime · Manages batching, sequences, evaluation, and prediction. · `src/evaluator/LlamaContext/LlamaContext.ts`
- [Token sampler](https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaContext/LlamaSampler.ts): decoding · `src/evaluator/LlamaContext/LlamaSampler.ts`
- [Chat session](https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaChatSession/LlamaChatSession.ts): conversation state · Turns prompt history into context evaluation. · `src/evaluator/LlamaChatSession/LlamaChatSession.ts`
- [Chat orchestration](https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaChat/LlamaChat.ts): function calling · `src/evaluator/LlamaChat/LlamaChat.ts`
- [Chat wrapper resolver](https://github.com/withcatai/node-llama-cpp/blob/master/src/chatWrappers/utils/resolveChatWrapper.ts): prompt formatting · Selects model-family or template-driven formatting. · `src/chatWrappers/utils/resolveChatWrapper.ts`
- [Structured grammar](https://github.com/withcatai/node-llama-cpp/blob/master/src/evaluator/LlamaJsonSchemaGrammar.ts): decoding constraints · Constrains JSON, schema, and function-call output. · `src/evaluator/LlamaJsonSchemaGrammar.ts`

### Models and GGUF

- [GGUF metadata reader](https://github.com/withcatai/node-llama-cpp/blob/master/src/gguf/readGgufFileInfo.ts): GGUF I/O · Reads local or fetched model metadata, tensors, and split files. · `src/gguf/readGgufFileInfo.ts`
- [GGUF parser](https://github.com/withcatai/node-llama-cpp/blob/master/src/gguf/parser/parseGguf.ts): format parser · `src/gguf/parser/parseGguf.ts`
- [Model downloader](https://github.com/withcatai/node-llama-cpp/blob/master/src/utils/createModelDownloader.ts): model acquisition · Resolves model URIs and downloads model files. · `src/utils/createModelDownloader.ts`

### Distribution and tooling

- [Platform binary packages](https://github.com/withcatai/node-llama-cpp/blob/master/packages/%40node-llama-cpp/linux-x64-cuda/src/index.ts): NPM binary distribution · Ship backend-specific prebuilt addons. · `packages/@node-llama-cpp/linux-x64-cuda/src/index.ts`
- [Project generator](https://github.com/withcatai/node-llama-cpp/blob/master/packages/create-node-llama-cpp/src/cli.ts): scaffolding CLI · `packages/create-node-llama-cpp/src/cli.ts`
- [Documentation site](https://github.com/withcatai/node-llama-cpp/blob/master/.vitepress/config.ts): VitePress · `.vitepress/config.ts`

## Connections

- Library barrel → Runtime initialization: initializes
- CLI dispatcher → Runtime initialization: initializes
- CLI dispatcher → Chat session: chat command
- CLI dispatcher → Model downloader: pull command
- Runtime initialization → Binary resolution: resolves binary
- Binary resolution → Compute backend selection: matches backend
- Binary resolution → Platform binary packages: loads prebuilt
- Binary resolution → Native compilation: fallback build
- Native compilation → CMake build: invokes
- CMake build → Node native addon: builds
- Platform binary packages → Node native addon: distribute
- Runtime initialization → Binding facade: returns
- Binding facade → Node native addon: calls
- Binding facade → Llama model: creates
- Llama model → GGUF metadata reader: loads model data
- GGUF metadata reader → GGUF parser: parses
- Model downloader → GGUF metadata reader: provides model file
- Llama model → Llama context: creates
- Llama context → Token sampler: samples tokens
- Chat session → Chat orchestration: orchestrates turns
- Chat session → Llama context: evaluates prompts
- Chat orchestration → Chat wrapper resolver: formats prompts
- Chat orchestration → Structured grammar: constrains output
- Structured grammar → Token sampler: guides decoding
- Project generator → Library barrel: scaffolds usage
- Documentation site → Library barrel: documents