How vahe1994/aqlm works
Project and its purpose
• AQLM is an open‐source command-line toolkit and library for quantizing and fine-tuning large language models (LLaMA, Mistral, Mixtral) via additive quantization (AQLM) and PV-Tuning.
• It provides:
• CLI entrypoints (main.py, finetune.py, lmeval.py, convert_to_hf.py, etc.)
• A core Python library (src/) implementing quantization, PV-Tuning, beam search, optimizers, data utilities.
• An inference library package (inference_lib/) with C++/CUDA, Triton, and Numba kernels.
• Benchmarking scripts (benchmark/) and Jupyter notebooks for demos.
• Integrations with Hugging Face Hub, Datasets, WandB, and Transformers.
Relationships and Data Flow
• User invokes a CLI script or notebook → script parses arguments → calls core library (src/aq.py, src/finetune.py, etc.)
• Core library downloads/unpacks model & dataset from HF Hub → performs quantization or PV-Tuning → writes quantized weights to disk
• For inference: core library or notebooks call inference_lib → kernel_selector picks fastest kernel → loads quantized weights → runs forward pass on GPU/CPU → returns generated tokens
• For benchmarks and evaluations: scripts call either core library or LM Evaluation Harness → aggregate metrics → optionally log to WandB
• Conversion script (convert_to_hf.py) reads quantized checkpoint → wraps it into a HF‐compatible transformers model → saves to output folder
Architectural patterns & design principles
• Modular layering: CLI → Orchestrator → Core quantization library → Inference kernels
• Plugin-style kernel selection: easily add new Triton/CUDA/Numba kernels under inference_kernels/
• Separation of concerns: quantization vs. fine-tuning vs. inference vs. evaluation
• Extensibility: can swap in new model families, kernel implementations, or evaluation tasks
Essential elements
1. CLI → Core
2. Core ↔ HF Hub (download/upload)
3. Core → Local Disk (save/load quantized model, tokenized data)
4. Core → Inference Library
5. Inference Library → Kernel Selector → [CUDA Kernel, Triton Kernel, Numba Kernel] → GPU/CPU
6. Scripts → WandB (optional)