How cornell-relaxml/yaqa works
Project Type and Purpose
• This is a research-grade Python library and CLI tool for quantizing large language models (LLaMA) using a Hessian-informed adaptive rounding algorithm (YAQA).
• It builds on the QTIP codebase, implements Hessian collection, quantization algorithms, GPU kernels, and evaluation scripts.
File Structure
Top-level directories indicate logical subsystems:
• eval/ ← evaluation scripts (KL, perplexity, zero-shot, interactive)
• hessian_llama/ ← modules to collect layerwise Hessians for LLaMA
• lib/ ← core Python library
– lib/algo/ ← quantization and finetuning algorithms
– lib/linear/ ← custom linear layers, quantized operators
– lib/codebook/ ← codebook routines (bit-shifting)
– lib/utils/ ← data loaders, math utilities, graph wrappers
• model/ ← high-level LLaMA model wrapper and caching utilities
• qtip-kernels/ ← custom CUDA/C++ kernels and Python bindings
• quantize_llama/ ← end-to-end scripts for quantization workflows
• assets/, README.md, requirements.txt
README for Additional Context
• Purpose: reduce KL divergence via Kronecker-factored Hessian approximations.
• Usage flow: collect Hessians → run quantization → evaluate.
• External dependency: HuggingFace model hub for pre-trained models.
Core Components & Their Relationships
a. Entry-Point Layer (CLI Scripts)
– quantize_llama/*.py
– eval/*.py
– hessian_llama/README.md points to usage of hessian scripts
↳ Orchestrates workflows and passes arguments/config to lower layers.
b. Model Adapter Layer
– model/llama.py & cache_utils.py
– hessian_llama/get_hess_llama.py & llama_hess.py
↳ Loads pre-trained LLaMA models; manages forward/backward passes to extract Hessians.
c. Core Library Layer (lib/)
– lib/algo (YAQA, LDLQ, finetune modules)
– lib/linear & lib/codebook (quantized operations)
– lib/utils (data handling, math, decomposition, sharding)
↳ Implements the quantization algorithm, data pipelines, utility functions.
d. GPU Kernel Layer (qtip-kernels/)
– inference.cu, qtip_torch.cu, wrapper.cpp
↳ Provides high-performance CUDA kernels; exposed to Python via setup.py bindings.
e. Data & Caching Layer
– model/cache_utils.py
– lib/utils/data_utils.py, gptq_data_utils.py
↳ Manages on-disk caches (Hessians, quantized parameters) and dataset iteration.
f. External Services
– HuggingFace Model Hub (download/upload pre-trained and quantized models)
– Optional: logging/metrics backends if added.
Architectural Patterns & Design Principles
• Layered Architecture:
CLI → Model Adapter → Core Algorithm Library → GPU Kernels → Hardware
• Plugin-Style Kernels: CUDA modules are isolated in qtip-kernels and loaded dynamically.
• Modularization by Concern:
– Algorithms vs. linear operators vs. I/O utilities vs. evaluation
• Data Flow:
– Raw model → Hessian collector → YAQA algorithm → quantized weights → evaluation