How cornell-relaxml/quip-sharp works
QuIP# is a weight-only post-training quantization system for compressing and deploying language models. Its main workflow prepares calibration Hessians, quantizes model layers with E8 lattice codebooks (optionally fine-tuning during quantization), saves a quantized model, and loads it for inference or evaluation. Separate capabilities include end-to-end tuning, Hugging Face format conversion, and perplexity, zero-shot, speed, and interactive-generation evaluation. Relationships among the quantization modules are supported by the sampled source index and excerpts; other stage links reflect README documentation, not inspection of unsampled implementations.
Main entry points: unsafe_import.py HF model loader · quantize_finetune_llama.py Layer quantization · hessian_offline_llama.py Hessian generation · quip.py QuIP algorithm
Components 26
Calibration Data
- Hessian generation
quantize_llama/hessian_offline_llama.py - Calibration utilities
lib/utils/data_utils.py - Hessian utilities
lib/utils/gptq_data_utils.py
Quantization
- Layer quantization
quantize_llama/quantize_finetune_llama.py - QuIP algorithm
lib/algo/quip.py - Quantization tuning
lib/algo/finetune.py - E8 codebooks
lib/codebook - E8P 2-bit
lib/codebook/latticee8_padded12.py - E8P RVQ 3-bit
lib/codebook/latticee8_padded12_rvq3bit.py - E8P RVQ 4-bit
lib/codebook/latticee8_padded12_rvq4bit.py - End-to-end tuning
quantize_llama/finetune_e2e_llama.py
Model Runtime
- Quantized linear
lib/linear/quantized_linear.py - Fused quantized linear
lib/linear/fused_quantized_linear.py - CUDA inference kernels
quiptools/quiptools_wrapper.cpp - Llama model
model/llama.py - Mistral model
model/mistral.py - HF model loader
lib/utils/unsafe_import.py - CUDA graph wrapper
lib/utils/graph_wrapper.py - HF model conversion
quantize_llama/hfize_llama.py
Evaluation
- Perplexity evaluation
eval/eval_ppl.py - Zero-shot evaluation
eval/eval_zeroshot.py - Speed evaluation
eval/eval_speed.py - Interactive generation
eval/interactive_gen.py - LM evaluation adapter
lib/utils/lm_eval_adaptor.py
Other components
- Model user
- Base HF model
Connections 38
- Model user → Hessian generation: prepares calibration
- Base HF model → Hessian generation: provides model
- Hessian generation → Hessian utilities: uses Hessian utilities
- Hessian generation → Calibration utilities: uses calibration data
- Model user → Layer quantization: runs quantization
- Base HF model → Layer quantization: provides weights
- Hessian generation → Layer quantization: supplies Hessians
- Layer quantization → QuIP algorithm: quantizes layers
quantize_llama/quantize_finetune_llama.py - Layer quantization → Quantization tuning: invokes tuning
quantize_llama/quantize_finetune_llama.py - Layer quantization → E8 codebooks: selects codebook
quantize_llama/quantize_finetune_llama.py - E8 codebooks → E8P 2-bit: registers
lib/codebook/__init__.py - E8 codebooks → E8P RVQ 3-bit: registers
lib/codebook/__init__.py - E8 codebooks → E8P RVQ 4-bit: registers
lib/codebook/__init__.py - QuIP algorithm → E8P 2-bit: quantizes with
- QuIP algorithm → E8P RVQ 3-bit: quantizes with
- QuIP algorithm → E8P RVQ 4-bit: quantizes with
- Quantization tuning → QuIP algorithm: calls
lib/algo/finetune.py - Quantization tuning → Quantized linear: instantiates
lib/algo/finetune.py - Quantized linear → E8 codebooks: selects implementation
lib/linear/quantized_linear.py - Fused quantized linear → Quantized linear: extends
lib/linear/fused_quantized_linear.py - Model user → HF model conversion: converts output
- Model user → End-to-end tuning: runs second tuning stage
- End-to-end tuning → HF model conversion: requires HF-converted model
- Model user → HF model loader: loads model
- HF model loader → Llama model: loads quantized Llama
lib/utils/unsafe_import.py - HF model loader → CUDA graph wrapper: optionally wraps
lib/utils/unsafe_import.py - Llama model → Fused quantized linear: uses layers
model/llama.py - Mistral model → Fused quantized linear: uses layers
model/mistral.py - Quantized linear → CUDA inference kernels: runs inference kernels
- Model user → Perplexity evaluation: evaluates perplexity
- Model user → Zero-shot evaluation: runs zero-shot tasks
- Model user → Speed evaluation: measures throughput
- Model user → Interactive generation: generates text
- Zero-shot evaluation → LM evaluation adapter: uses evaluation adapter
- Perplexity evaluation → HF model loader: loads model
eval/eval_ppl.py - Zero-shot evaluation → HF model loader: loads model
eval/eval_zeroshot.py - Speed evaluation → HF model loader: loads model
eval/eval_speed.py - Interactive generation → HF model loader: loads model
eval/interactive_gen.py