How leanmodels/dfloat11 works
Project type and purpose
• This is a Python library + CLI tool for lossless compression of LLM weights (“DFloat11”) and fast GPU inference.
• It integrates with Hugging Face Transformers and Accelerate to load a standard BFloat16 model and a corresponding compressed DFloat11 model, then runs causal-LM generation on CUDA devices.
File structure
• Top level:
• setup.py – package installation
• inference.py – CLI entry point for inference
• dfloat11/ – the core library
– modeling_dfloat11_llama.py – defines DFloat11ModelForCausalLM, weight-loading logic, and custom CUDA kernel hooks
• No microservices or separate frontend; this is a standalone tool/library.
• Dependencies (in README): PyTorch, Hugging Face Transformers & Hub, Accelerate, CUDA.
README for additional insights
• Describes a single-process pipeline: download model weights, instantiate a Hugging Face model wrapped by DFloat11 loader, dispatch to GPU(s) via Accelerate, run generate loops.
• Highlights the compression ratio (~30%), bit-identical outputs, and custom GPU kernel for decompression & forward pass.