# cornell-relaxml/quip-sharp architecture

> Architecture diagram of the GitHub repository cornell-relaxml/quip-sharp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cornell-relaxml/quip-sharp
- Repository: https://github.com/cornell-relaxml/quip-sharp

## Overview

QuIP# is a weight-only post-training quantization system for compressing and deploying language models. Its main workflow prepares calibration Hessians, quantizes model layers with E8 lattice codebooks (optionally fine-tuning during quantization), saves a quantized model, and loads it for inference or evaluation. Separate capabilities include end-to-end tuning, Hugging Face format conversion, and perplexity, zero-shot, speed, and interactive-generation evaluation. Relationships among the quantization modules are supported by the sampled source index and excerpts; other stage links reflect README documentation, not inspection of unsampled implementations.

Main entry points: [`lib/utils/unsafe_import.py`](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/utils/unsafe_import.py) (HF model loader), [`quantize_llama/quantize_finetune_llama.py`](https://github.com/cornell-relaxml/quip-sharp/blob/main/quantize_llama/quantize_finetune_llama.py) (Layer quantization), [`quantize_llama/hessian_offline_llama.py`](https://github.com/cornell-relaxml/quip-sharp/blob/main/quantize_llama/hessian_offline_llama.py) (Hessian generation), [`lib/algo/quip.py`](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/algo/quip.py) (QuIP algorithm)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Calibration Data"]
  node_hessian["Hessian generation"]
  node_data["Calibration utilities<br/>[data_utils.py]"]
  node_gptqdata["Hessian utilities<br/>[gptq_data_utils.py]"]
end

subgraph group_quant["Quantization"]
  node_quantflow["Layer quantization"]
  node_quip["QuIP algorithm<br/>[quip.py]"]
  node_finetune["Quantization tuning<br/>[finetune.py]"]
  node_codebooks["E8 codebooks"]
  node_cb2["E8P 2-bit"]
  node_cb3["E8P RVQ 3-bit"]
  node_cb4["E8P RVQ 4-bit"]
  node_e2e["End-to-end tuning"]
end

subgraph group_deploy["Model Runtime"]
  node_linear["Quantized linear"]
  node_fusedlinear["Fused quantized linear"]
  node_kernels["CUDA inference kernels"]
  node_llama["Llama model<br/>[llama.py]"]
  node_mistral["Mistral model<br/>[mistral.py]"]
  node_loader["HF model loader<br/>[unsafe_import.py]"]
  node_graph["CUDA graph wrapper<br/>[graph_wrapper.py]"]
  node_hfize["HF model conversion<br/>[hfize_llama.py]"]
end

subgraph group_eval["Evaluation"]
  node_ppl["Perplexity evaluation<br/>[eval_ppl.py]"]
  node_zero["Zero-shot evaluation<br/>[eval_zeroshot.py]"]
  node_speed["Speed evaluation<br/>[eval_speed.py]"]
  node_interactive["Interactive generation<br/>[interactive_gen.py]"]
  node_adapter["LM evaluation adapter<br/>[lm_eval_adaptor.py]"]
end

node_user(("Model user"))
node_hfbase["Base HF model"]

node_user -->|"prepares calibration"| node_hessian
node_hfbase -->|"provides model"| node_hessian
node_hessian -->|"uses Hessian utilities"| node_gptqdata
node_hessian -.->|"uses calibration data"| node_data
node_user -->|"runs quantization"| node_quantflow
node_hfbase -->|"provides weights"| node_quantflow
node_hessian -->|"supplies Hessians"| node_quantflow
node_quantflow -->|"quantizes layers"| node_quip
node_quantflow -->|"invokes tuning"| node_finetune
node_quantflow -->|"selects codebook"| node_codebooks
node_codebooks -->|"registers"| node_cb2
node_codebooks -->|"registers"| node_cb3
node_codebooks -->|"registers"| node_cb4
node_quip -.->|"quantizes with"| node_cb2
node_quip -.->|"quantizes with"| node_cb3
node_quip -.->|"quantizes with"| node_cb4
node_finetune -->|"calls"| node_quip
node_finetune -->|"instantiates"| node_linear
node_linear -->|"selects implementation"| node_codebooks
node_fusedlinear -->|"extends"| node_linear
node_user -->|"converts output"| node_hfize
node_user -->|"runs second tuning stage"| node_e2e
node_e2e -->|"requires HF-converted model"| node_hfize
node_user -->|"loads model"| node_loader
node_loader -->|"loads quantized Llama"| node_llama
node_loader -->|"optionally wraps"| node_graph
node_llama -->|"uses layers"| node_fusedlinear
node_mistral -->|"uses layers"| node_fusedlinear
node_linear -.->|"runs inference kernels"| node_kernels
node_user -->|"evaluates perplexity"| node_ppl
node_user -->|"runs zero-shot tasks"| node_zero
node_user -->|"measures throughput"| node_speed
node_user -->|"generates text"| node_interactive
node_zero -.->|"uses evaluation adapter"| node_adapter
node_ppl -->|"loads model"| node_loader
node_zero -->|"loads model"| node_loader
node_speed -->|"loads model"| node_loader
node_interactive -->|"loads model"| node_loader

click node_hessian "https://github.com/cornell-relaxml/quip-sharp/blob/main/quantize_llama/hessian_offline_llama.py"
click node_data "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/utils/data_utils.py"
click node_gptqdata "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/utils/gptq_data_utils.py"
click node_quantflow "https://github.com/cornell-relaxml/quip-sharp/blob/main/quantize_llama/quantize_finetune_llama.py"
click node_quip "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/algo/quip.py"
click node_finetune "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/algo/finetune.py"
click node_codebooks "https://github.com/cornell-relaxml/quip-sharp/tree/main/lib/codebook"
click node_cb2 "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/codebook/latticee8_padded12.py"
click node_cb3 "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/codebook/latticee8_padded12_rvq3bit.py"
click node_cb4 "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/codebook/latticee8_padded12_rvq4bit.py"
click node_linear "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/linear/quantized_linear.py"
click node_fusedlinear "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/linear/fused_quantized_linear.py"
click node_kernels "https://github.com/cornell-relaxml/quip-sharp/blob/main/quiptools/quiptools_wrapper.cpp"
click node_llama "https://github.com/cornell-relaxml/quip-sharp/blob/main/model/llama.py"
click node_mistral "https://github.com/cornell-relaxml/quip-sharp/blob/main/model/mistral.py"
click node_loader "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/utils/unsafe_import.py"
click node_graph "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/utils/graph_wrapper.py"
click node_hfize "https://github.com/cornell-relaxml/quip-sharp/blob/main/quantize_llama/hfize_llama.py"
click node_e2e "https://github.com/cornell-relaxml/quip-sharp/blob/main/quantize_llama/finetune_e2e_llama.py"
click node_ppl "https://github.com/cornell-relaxml/quip-sharp/blob/main/eval/eval_ppl.py"
click node_zero "https://github.com/cornell-relaxml/quip-sharp/blob/main/eval/eval_zeroshot.py"
click node_speed "https://github.com/cornell-relaxml/quip-sharp/blob/main/eval/eval_speed.py"
click node_interactive "https://github.com/cornell-relaxml/quip-sharp/blob/main/eval/interactive_gen.py"
click node_adapter "https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/utils/lm_eval_adaptor.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_hessian,node_data,node_gptqdata,node_user toneBlue
class node_quantflow,node_quip,node_finetune,node_codebooks,node_cb2,node_cb3,node_cb4,node_e2e toneAmber
class node_linear,node_fusedlinear,node_kernels,node_llama,node_mistral,node_loader,node_graph,node_hfize toneMint
class node_ppl,node_zero,node_speed,node_interactive,node_adapter toneRose
class node_hfbase toneIndigo
```

## Components

### Calibration Data

- [Hessian generation](https://github.com/cornell-relaxml/quip-sharp/blob/main/quantize_llama/hessian_offline_llama.py): `quantize_llama/hessian_offline_llama.py`
- [Calibration utilities](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/utils/data_utils.py): `lib/utils/data_utils.py`
- [Hessian utilities](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/utils/gptq_data_utils.py): `lib/utils/gptq_data_utils.py`

### Quantization

- [Layer quantization](https://github.com/cornell-relaxml/quip-sharp/blob/main/quantize_llama/quantize_finetune_llama.py): `quantize_llama/quantize_finetune_llama.py`
- [QuIP algorithm](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/algo/quip.py): `lib/algo/quip.py`
- [Quantization tuning](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/algo/finetune.py): `lib/algo/finetune.py`
- [E8 codebooks](https://github.com/cornell-relaxml/quip-sharp/tree/main/lib/codebook): `lib/codebook`
- [E8P 2-bit](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/codebook/latticee8_padded12.py): `lib/codebook/latticee8_padded12.py`
- [E8P RVQ 3-bit](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/codebook/latticee8_padded12_rvq3bit.py): `lib/codebook/latticee8_padded12_rvq3bit.py`
- [E8P RVQ 4-bit](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/codebook/latticee8_padded12_rvq4bit.py): `lib/codebook/latticee8_padded12_rvq4bit.py`
- [End-to-end tuning](https://github.com/cornell-relaxml/quip-sharp/blob/main/quantize_llama/finetune_e2e_llama.py): `quantize_llama/finetune_e2e_llama.py`

### Model Runtime

- [Quantized linear](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/linear/quantized_linear.py): `lib/linear/quantized_linear.py`
- [Fused quantized linear](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/linear/fused_quantized_linear.py): `lib/linear/fused_quantized_linear.py`
- [CUDA inference kernels](https://github.com/cornell-relaxml/quip-sharp/blob/main/quiptools/quiptools_wrapper.cpp): `quiptools/quiptools_wrapper.cpp`
- [Llama model](https://github.com/cornell-relaxml/quip-sharp/blob/main/model/llama.py): `model/llama.py`
- [Mistral model](https://github.com/cornell-relaxml/quip-sharp/blob/main/model/mistral.py): `model/mistral.py`
- [HF model loader](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/utils/unsafe_import.py): `lib/utils/unsafe_import.py`
- [CUDA graph wrapper](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/utils/graph_wrapper.py): `lib/utils/graph_wrapper.py`
- [HF model conversion](https://github.com/cornell-relaxml/quip-sharp/blob/main/quantize_llama/hfize_llama.py): `quantize_llama/hfize_llama.py`

### Evaluation

- [Perplexity evaluation](https://github.com/cornell-relaxml/quip-sharp/blob/main/eval/eval_ppl.py): `eval/eval_ppl.py`
- [Zero-shot evaluation](https://github.com/cornell-relaxml/quip-sharp/blob/main/eval/eval_zeroshot.py): `eval/eval_zeroshot.py`
- [Speed evaluation](https://github.com/cornell-relaxml/quip-sharp/blob/main/eval/eval_speed.py): `eval/eval_speed.py`
- [Interactive generation](https://github.com/cornell-relaxml/quip-sharp/blob/main/eval/interactive_gen.py): `eval/interactive_gen.py`
- [LM evaluation adapter](https://github.com/cornell-relaxml/quip-sharp/blob/main/lib/utils/lm_eval_adaptor.py): `lib/utils/lm_eval_adaptor.py`

### Other components

- **Model user**
- **Base HF model**

## Connections

- Model user → Hessian generation: prepares calibration
- Base HF model → Hessian generation: provides model
- Hessian generation → Hessian utilities: uses Hessian utilities
- Hessian generation → Calibration utilities: uses calibration data
- Model user → Layer quantization: runs quantization
- Base HF model → Layer quantization: provides weights
- Hessian generation → Layer quantization: supplies Hessians
- Layer quantization → QuIP algorithm: quantizes layers (evidence: `quantize_llama/quantize_finetune_llama.py`)
- Layer quantization → Quantization tuning: invokes tuning (evidence: `quantize_llama/quantize_finetune_llama.py`)
- Layer quantization → E8 codebooks: selects codebook (evidence: `quantize_llama/quantize_finetune_llama.py`)
- E8 codebooks → E8P 2-bit: registers (evidence: `lib/codebook/__init__.py`)
- E8 codebooks → E8P RVQ 3-bit: registers (evidence: `lib/codebook/__init__.py`)
- E8 codebooks → E8P RVQ 4-bit: registers (evidence: `lib/codebook/__init__.py`)
- QuIP algorithm → E8P 2-bit: quantizes with
- QuIP algorithm → E8P RVQ 3-bit: quantizes with
- QuIP algorithm → E8P RVQ 4-bit: quantizes with
- Quantization tuning → QuIP algorithm: calls (evidence: `lib/algo/finetune.py`)
- Quantization tuning → Quantized linear: instantiates (evidence: `lib/algo/finetune.py`)
- Quantized linear → E8 codebooks: selects implementation (evidence: `lib/linear/quantized_linear.py`)
- Fused quantized linear → Quantized linear: extends (evidence: `lib/linear/fused_quantized_linear.py`)
- Model user → HF model conversion: converts output
- Model user → End-to-end tuning: runs second tuning stage
- End-to-end tuning → HF model conversion: requires HF-converted model
- Model user → HF model loader: loads model
- HF model loader → Llama model: loads quantized Llama (evidence: `lib/utils/unsafe_import.py`)
- HF model loader → CUDA graph wrapper: optionally wraps (evidence: `lib/utils/unsafe_import.py`)
- Llama model → Fused quantized linear: uses layers (evidence: `model/llama.py`)
- Mistral model → Fused quantized linear: uses layers (evidence: `model/mistral.py`)
- Quantized linear → CUDA inference kernels: runs inference kernels
- Model user → Perplexity evaluation: evaluates perplexity
- Model user → Zero-shot evaluation: runs zero-shot tasks
- Model user → Speed evaluation: measures throughput
- Model user → Interactive generation: generates text
- Zero-shot evaluation → LM evaluation adapter: uses evaluation adapter
- Perplexity evaluation → HF model loader: loads model (evidence: `eval/eval_ppl.py`)
- Zero-shot evaluation → HF model loader: loads model (evidence: `eval/eval_zeroshot.py`)
- Speed evaluation → HF model loader: loads model (evidence: `eval/eval_speed.py`)
- Interactive generation → HF model loader: loads model (evidence: `eval/interactive_gen.py`)
