# kevbuh/bitnet architecture

> Architecture diagram of the GitHub repository kevbuh/bitnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kevbuh/bitnet
- Repository: https://github.com/kevbuh/bitnet

## Overview

##### Project type and purpose
- bitnet is an open-source machine-learning library and toolkit for defining, training, and running a binarized large language model (“BitNet”), based on LLaMa
- It provides:
– Core model definition (ternary-weight STE, 8-bit activations, SubLN, ReLU², RoPE/GQA attention)
– Training pipeline
– Utility functions (quantization, weight management)
– Pre-trained model weights for inference
– Test suite and CI

##### File structure
- Top level:
– .github/workflows/ci.yml → GitHub Actions CI for linting/tests
– setup.sh → environment & dependency setup (venv, torch, jax, etc.)
– SPEC.md → specification of binary kernels, hardware targets, performance goals
– bitnet/ → Python package with:
- model.py → model class definitions and loading config/weights
- train.py → end-to-end training script (data loader, optimizer, loss, STE quant)
- utils.py → quantization functions, helper routines
- __init__.py → package entry point
- bitnet_wikitext2_1_9500.pt → sample pre-trained weights file

##### README for additional insights
- Explains model architecture (config.json fields), quantization strategy (scale & round to –1,0,+1), references to academic papers
- Describes future directions (binary kernels, FPGA, MoE)

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Data & Config"
        D1["data/input.txt"]:::data
        D2["data/layer_shapes.txt"]:::data
        D3["SPEC.md"]:::data
        D4["config.json"]:::data
    end

    subgraph "bitnet Package"
        M1["model.py: BitNetForCausalLM"]:::code
        M2["__init__.py"]:::code
        M3["train.py"]:::code
        M4["utils.py: Quant & Helpers"]:::code
        W1["bitnet_wikitext2_1_9500.pt"]:::data
    end

    E1["PyTorch / HuggingFace"]:::external
    E2["JAX (optional)"]:::external
    CI["GitHub Actions CI"]:::cicd

    subgraph "Tests"
        T1["tests/test_layers.py"]:::code
        T2["tests/test_quant.py"]:::code
    end

    Deploy["setup.sh"]:::code

    %% Connections
    D1 -->|feeds sample text| M3
    D2 -->|feeds shapes| M3
    D3 -->|reference specs| M3
    D4 -->|loads config| M1

    M3 -->|imports core classes| M1
    M3 -->|imports quant funcs| M4
    M4 -->|provides helpers| M1
    M4 -->|provides helpers| M3
    W1 -->|loads weights| M1

    E1 -->|runtime| M1
    E1 -->|runtime| M3
    E2 -->|optional backend| M1
    E2 -->|optional backend| M3

    CI -->|runs tests| T1
    CI -->|runs tests| T2
    T1 -->|validates| M1
    T2 -->|validates| M4

    Deploy -->|sets up env| M3
    Deploy -->|sets up env| M1

    %% Legend
    subgraph Legend
        C1["Code Modules"]:::code
        Dd["Data & Config"]:::data
        Ex["External Dependencies"]:::external
        Ci["CI/CD & Tests"]:::cicd
    end

    %% Click Events
    click D1 "https://github.com/kevbuh/bitnet/blob/main/data/input.txt"
    click D2 "https://github.com/kevbuh/bitnet/blob/main/data/layer_shapes.txt"
    click D3 "https://github.com/kevbuh/bitnet/blob/main/SPEC.md"
    click M1 "https://github.com/kevbuh/bitnet/blob/main/bitnet/model.py"
    click M2 "https://github.com/kevbuh/bitnet/blob/main/bitnet/__init__.py"
    click M3 "https://github.com/kevbuh/bitnet/blob/main/bitnet/train.py"
    click M4 "https://github.com/kevbuh/bitnet/blob/main/bitnet/utils.py"
    click W1 "https://github.com/kevbuh/bitnet/blob/main/bitnet_wikitext2_1_9500.pt"
    click CI "https://github.com/kevbuh/bitnet/blob/main/.github/workflows/ci.yml"
    click T1 "https://github.com/kevbuh/bitnet/blob/main/tests/test_layers.py"
    click T2 "https://github.com/kevbuh/bitnet/blob/main/tests/test_quant.py"
    click Deploy "https://github.com/kevbuh/bitnet/blob/main/setup.sh"

    %% Styles
    classDef code fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef data fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px
    classDef cicd fill:#DDA0DD,stroke:#333,stroke-width:1px
```
