# microsoft/bitnet architecture

> Architecture diagram of the GitHub repository microsoft/bitnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/bitnet
- Repository: https://github.com/microsoft/bitnet

## Overview

bitnet.cpp is an inference framework for ternary/1-bit language and embedding models, with optimized CPU kernels and a separate GPU inference path. The user workflow is to prepare a compatible model, then run tokenized prompts through model inference and receive generated output; CPU inference is integrated with the bundled llama.cpp tree, while GPU generation has its own Python runtime. The graph preserves model conversion, embedding support, and CPU kernel variants. Some runtime wiring is unsampled, so uncertain links are omitted rather than inferred.

Main entry points: [`gpu/generate.py`](https://github.com/microsoft/bitnet/blob/main/gpu/generate.py) (GPU Generation), [`src/ggml-bitnet-mad.cpp`](https://github.com/microsoft/bitnet/blob/main/src/ggml-bitnet-mad.cpp) (BitNet Operations), [`src/ggml-bitnet-lut.cpp`](https://github.com/microsoft/bitnet/blob/main/src/ggml-bitnet-lut.cpp) (LUT Operations), [`gpu/convert_checkpoint.py`](https://github.com/microsoft/bitnet/blob/main/gpu/convert_checkpoint.py) (GPU Checkpoint Prep)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prepare["Model Preparation"]
  node_hfconvert["HF Conversion"]
  node_conversion["Conversion Tools"]
  node_embedding["Embedding Conversion"]
end

subgraph group_cpu["CPU Inference"]
  node_cpuentry["CPU Runtime<br/>[llama.cpp]"]
  node_cpuops["BitNet Operations"]
  node_lutops["LUT Operations"]
  node_bitnetapi["BitNet API<br/>[ggml-bitnet.h]"]
end

subgraph group_gpu["GPU Inference"]
  node_gpuconvert["GPU Checkpoint Prep"]
  node_weightpack["Weight Packing<br/>[pack_weight.py]"]
  node_gpumodel["GPU Transformer<br/>[model.py]"]
  node_generation["GPU Generation<br/>[generate.py]"]
  node_tokenizer["Tokenizer and Chat<br/>[tokenizer.py]"]
  node_sampling["Sampling<br/>[sample_utils.py]"]
  node_generationstats["Generation Stats<br/>[stats.py]"]
end

subgraph group_kernels["Kernel Implementations"]
  node_lutkernels["LUT Kernels"]
  node_presetkernels["Preset Kernels"]
end

node_user(("Model User"))

node_user -->|"prepares model"| node_hfconvert
node_user -->|"runs inference"| node_cpuentry
node_user -->|"prompts generation"| node_generation
node_hfconvert -->|"uses conversion"| node_conversion
node_gpuconvert -->|"packs weights"| node_weightpack
node_gpuconvert -->|"uses model"| node_gpumodel
node_generation -->|"runs transformer"| node_gpumodel
node_generation -->|"encodes prompts"| node_tokenizer
node_generation -->|"uses sampling"| node_sampling
node_generation -->|"records stats"| node_generationstats
node_cpuops -->|"uses API"| node_bitnetapi
node_lutops -->|"uses API"| node_bitnetapi
node_lutops -->|"includes kernels"| node_lutkernels
node_cpuops -.->|"selects variants"| node_presetkernels
node_user -->|"converts embeddings"| node_embedding

click node_hfconvert "https://github.com/microsoft/bitnet/blob/main/utils/convert-hf-to-gguf-bitnet.py"
click node_conversion "https://github.com/microsoft/bitnet/tree/main/utils"
click node_embedding "https://github.com/microsoft/bitnet/blob/main/utils/convert-bitnet-embedding-to-gguf.py"
click node_cpuentry "https://github.com/microsoft/bitnet/blob/main/3rdparty/llama.cpp"
click node_cpuops "https://github.com/microsoft/bitnet/blob/main/src/ggml-bitnet-mad.cpp"
click node_lutops "https://github.com/microsoft/bitnet/blob/main/src/ggml-bitnet-lut.cpp"
click node_bitnetapi "https://github.com/microsoft/bitnet/blob/main/include/ggml-bitnet.h"
click node_gpuconvert "https://github.com/microsoft/bitnet/blob/main/gpu/convert_checkpoint.py"
click node_weightpack "https://github.com/microsoft/bitnet/blob/main/gpu/pack_weight.py"
click node_gpumodel "https://github.com/microsoft/bitnet/blob/main/gpu/model.py"
click node_generation "https://github.com/microsoft/bitnet/blob/main/gpu/generate.py"
click node_tokenizer "https://github.com/microsoft/bitnet/blob/main/gpu/tokenizer.py"
click node_sampling "https://github.com/microsoft/bitnet/blob/main/gpu/sample_utils.py"
click node_generationstats "https://github.com/microsoft/bitnet/blob/main/gpu/stats.py"
click node_lutkernels "https://github.com/microsoft/bitnet/blob/main/include/bitnet-lut-kernels.h"
click node_presetkernels "https://github.com/microsoft/bitnet/tree/main/preset_kernels"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_hfconvert,node_conversion,node_embedding,node_user toneBlue
class node_cpuentry,node_cpuops,node_lutops,node_bitnetapi toneAmber
class node_gpuconvert,node_weightpack,node_gpumodel,node_generation,node_tokenizer,node_sampling,node_generationstats toneMint
class node_lutkernels,node_presetkernels toneRose
```

## Components

### Model Preparation

- [HF Conversion](https://github.com/microsoft/bitnet/blob/main/utils/convert-hf-to-gguf-bitnet.py): `utils/convert-hf-to-gguf-bitnet.py`
- [Conversion Tools](https://github.com/microsoft/bitnet/tree/main/utils): `utils`
- [Embedding Conversion](https://github.com/microsoft/bitnet/blob/main/utils/convert-bitnet-embedding-to-gguf.py): `utils/convert-bitnet-embedding-to-gguf.py`

### CPU Inference

- [CPU Runtime](https://github.com/microsoft/bitnet/blob/main/3rdparty/llama.cpp): `3rdparty/llama.cpp`
- [BitNet Operations](https://github.com/microsoft/bitnet/blob/main/src/ggml-bitnet-mad.cpp): `src/ggml-bitnet-mad.cpp`
- [LUT Operations](https://github.com/microsoft/bitnet/blob/main/src/ggml-bitnet-lut.cpp): `src/ggml-bitnet-lut.cpp`
- [BitNet API](https://github.com/microsoft/bitnet/blob/main/include/ggml-bitnet.h): `include/ggml-bitnet.h`

### GPU Inference

- [GPU Checkpoint Prep](https://github.com/microsoft/bitnet/blob/main/gpu/convert_checkpoint.py): `gpu/convert_checkpoint.py`
- [Weight Packing](https://github.com/microsoft/bitnet/blob/main/gpu/pack_weight.py): `gpu/pack_weight.py`
- [GPU Transformer](https://github.com/microsoft/bitnet/blob/main/gpu/model.py): `gpu/model.py`
- [GPU Generation](https://github.com/microsoft/bitnet/blob/main/gpu/generate.py): `gpu/generate.py`
- [Tokenizer and Chat](https://github.com/microsoft/bitnet/blob/main/gpu/tokenizer.py): `gpu/tokenizer.py`
- [Sampling](https://github.com/microsoft/bitnet/blob/main/gpu/sample_utils.py): `gpu/sample_utils.py`
- [Generation Stats](https://github.com/microsoft/bitnet/blob/main/gpu/stats.py): `gpu/stats.py`

### Kernel Implementations

- [LUT Kernels](https://github.com/microsoft/bitnet/blob/main/include/bitnet-lut-kernels.h): `include/bitnet-lut-kernels.h`
- [Preset Kernels](https://github.com/microsoft/bitnet/tree/main/preset_kernels): `preset_kernels`

### Other components

- **Model User**

## Connections

- Model User → HF Conversion: prepares model (evidence: `README.md`)
- Model User → CPU Runtime: runs inference (evidence: `README.md`)
- Model User → GPU Generation: prompts generation (evidence: `gpu/generate.py`)
- HF Conversion → Conversion Tools: uses conversion (evidence: `utils/convert-hf-to-gguf-bitnet.py`)
- GPU Checkpoint Prep → Weight Packing: packs weights (evidence: `gpu/convert_checkpoint.py`)
- GPU Checkpoint Prep → GPU Transformer: uses model (evidence: `gpu/convert_checkpoint.py`)
- GPU Generation → GPU Transformer: runs transformer (evidence: `gpu/generate.py`)
- GPU Generation → Tokenizer and Chat: encodes prompts (evidence: `gpu/generate.py`)
- GPU Generation → Sampling: uses sampling (evidence: `gpu/generate.py`)
- GPU Generation → Generation Stats: records stats (evidence: `gpu/generate.py`)
- BitNet Operations → BitNet API: uses API (evidence: `src/ggml-bitnet-mad.cpp`)
- LUT Operations → BitNet API: uses API (evidence: `src/ggml-bitnet-lut.cpp`)
- LUT Operations → LUT Kernels: includes kernels (evidence: `src/ggml-bitnet-lut.cpp`)
- BitNet Operations → Preset Kernels: selects variants
- Model User → Embedding Conversion: converts embeddings (evidence: `README.md`)
