How kevbuh/bitnet works
Project type and purpose
• bitnet is an open-source machine-learning library and toolkit for defining, training, and running a binarized large language model (“BitNet”), based on LLaMa
• It provides:
– Core model definition (ternary-weight STE, 8-bit activations, SubLN, ReLU², RoPE/GQA attention)
– Training pipeline
– Utility functions (quantization, weight management)
– Pre-trained model weights for inference
– Test suite and CI
File structure
• Top level:
– .github/workflows/ci.yml → GitHub Actions CI for linting/tests
– setup.sh → environment & dependency setup (venv, torch, jax, etc.)
– SPEC.md → specification of binary kernels, hardware targets, performance goals
– bitnet/ → Python package with:
• model.py → model class definitions and loading config/weights
• train.py → end-to-end training script (data loader, optimizer, loss, STE quant)
• utils.py → quantization functions, helper routines
• __init__.py → package entry point
• bitnet_wikitext2_1_9500.pt → sample pre-trained weights file
README for additional insights
• Explains model architecture (config.json fields), quantization strategy (scale & round to –1,0,+1), references to academic papers
• Describes future directions (binary kernels, FPGA, MoE)