withcatai/node-llama-cpp

Purpose and entry points

node-llama-cpp is a TypeScript/Node.js SDK and CLI for local GGUF LLM inference through native llama.cpp bindings. The public library barrel is src/index.ts; runtime initialization begins with src/bindings/getLlama.ts. The terminal interface starts at src/cli/cli.ts and dispatches commands such as chat, completion, model download, inspection, and source builds.

Native binding and runtime selection

The core boundary is TypeScript-to-C++ Node addon. llama/addon/addon.cpp registers native exports; object wrappers such as llama/addon/AddonModel.cpp and llama/addon/AddonGrammar.cpp own native model and constraint state. TypeScript binding facades in src/bindings/Llama.ts and src/bindings/AddonTypes.ts expose this functionality safely to the evaluator layer.

At startup, binding resolution selects compatible prebuilt platform packages or falls back to obtaining and compiling llama.cpp. This path is coordinated by src/bindings/getLlama.ts, src/bindings/utils/resolveActualBindingBinaryPath.ts, and src/bindings/utils/compileLLamaCpp.ts.

Hardware, builds, and binary distribution

Compute backend selection detects available CPU/GPU capabilities and chooses Metal, CUDA, or Vulkan variants where applicable. Key decision points are src/bindings/utils/detectAvailableComputeLayers.ts, src/bindings/utils/getBestComputeLayersAvailable.ts, and native Vulkan probing in llama/gpuInfo/vulkan-gpu-info.cpp.

CMake drives native compilation through llama/CMakeLists.txt, with cross-platform toolchains under llama/toolchains/. Standalone NPM packages distribute platform/backend-specific binaries, e.g. packages/@node-llama-cpp/linux-x64-cuda/src/index.ts and packages/@node-llama-cpp/mac-arm64-metal/src/index.ts.

Model lifecycle and inference

Application flow is: initialize Llama → load a GGUF model → create a context/sequences → tokenize/evaluate/sample generated tokens → dispose native resources. Main abstractions are src/evaluator/LlamaModel/LlamaModel.ts, src/evaluator/LlamaContext/LlamaContext.ts, and src/evaluator/LlamaContext/LlamaSampler.ts. The context layer manages batching, parallel work prioritization, checkpoints, token prediction, and memory-sensitive inference boundaries.

Chat orchestration and structured generation

LlamaChatSession maintains history and turns prompts into context evaluation, while LlamaChat supplies function-calling and response orchestration. Primary paths: src/evaluator/LlamaChatSession/LlamaChatSession.ts, src/evaluator/LlamaChat/LlamaChat.ts, and src/ChatWrapper.ts.

Model-family chat formatting is isolated behind specialized and template-driven wrappers, resolved by src/chatWrappers/utils/resolveChatWrapper.ts; generic templates are handled in src/chatWrappers/generic/JinjaTemplateChatWrapper.ts. Grammar classes constrain decoding for JSON/schema and function-call output: src/evaluator/LlamaGrammar.ts and src/evaluator/LlamaJsonSchemaGrammar.ts.

GGUF I/O and model acquisition

The GGUF subsystem parses local or fetched model metadata, tensors, split files, and resource estimates independently of model loading. Its boundary begins at src/gguf/readGgufFileInfo.ts, with version parsers in src/gguf/parser/parseGguf.ts and readers including src/gguf/fileReaders/GgufNetworkFetchFileReader.ts.

CLI/model utilities resolve model URIs and download destinations, including GitHub/Hugging Face-style endpoints, via src/utils/createModelDownloader.ts, src/utils/parseModelUri.ts, and src/utils/modelDownloadEndpoints.ts.

CLI, scaffolding, and documentation

CLI commands are an alternative integration surface over the same library lifecycle: src/cli/commands/ChatCommand.ts, src/cli/commands/PullCommand.ts, and src/cli/commands/source/SourceCommand.ts. Project generation is separately packaged in packages/create-node-llama-cpp/src/cli.ts; templates include Node and Electron applications. VitePress documentation is built from docs/ and configured at .vitepress/config.ts.