How vllm-project/llm-compressor works
LLM Compressor is a Python library for post-training model compression, mainly for Hugging Face transformer models, with outputs intended to run in vLLM. The repository is organized around a small core engine plus algorithm-specific modifiers, model-specific support code, and many end-to-end examples/tests.
1. Top-level runtime flow
The main user-facing path is: load a model → choose a compression recipe/modifier → run an entrypoint like oneshot or model_free_ptq → apply calibration/analysis/rewrites → save a compressed checkpoint in a vLLM-compatible safetensors-based format.
2. Public entrypoints
src/llmcompressor/entrypoints/oneshot.py is the standard one-shot compression API.
src/llmcompressor/entrypoints/model_free/ is a separate pathway for model-free PTQ, used when Hugging Face model definitions are unavailable or unreliable. It has its own lifecycle, validation, reindexing, save utilities, and process orchestration.
3. Recipe and modifier abstraction
Compression is driven by recipe objects and Modifier implementations. The recipe package defines the configuration/metadata layer, while modifiers/ contains the actual algorithm behaviors. This is the central extensibility boundary: recipes describe what to do; modifiers implement how to do it.
4. Lifecycle and state management
The core/ package provides session/state/event/lifecycle coordination. This is the control plane that sequences compression steps, tracks model state, and emits events across the pipeline. It keeps orchestration separate from algorithm code.
5. Compression algorithm families
The modifiers are grouped by technique:
- quantization/ for PTQ-style quantization behaviors
- gptq/, awq/, autoround/, smoothquant/, obcq/
- pruning/ for sparsity-related workflows
- transform/ for model rewrites and layout changes such as AWQ, SpinQuant, QuIP, iMatrix, SmoothQuant transforms
- experimental/ modifiers for newer or less stable paths
This structure reflects a plugin-like architecture where algorithm families share common interfaces but have specialized logic.
6. Observers and calibration
observers/ contains measurement components such as min/max, MSE, moving statistics, and iMatrix observers. These collect activation or weight statistics during calibration passes and feed quantization decisions. This is the measurement layer used by multiple compression schemes.
7. Pipelines as execution strategies
pipelines/ contains the orchestration strategies: basic, data-free, independent, and sequential. These determine how modifiers are run, in what order, and how model passes are coordinated. The sequential pipeline includes AST helpers because it needs to analyze or wrap model execution paths for calibration/inference tracing.
8. Modeling and model-specific support
modeling/ holds architecture-specific calibration/fusion/patch logic for families like Llama 4, Qwen3, Gemma4, DeepSeek V3/V3.2, Granite4, GLM MoE, and Qwen multimodal/MoE variants. This is where the repo adapts generic compression logic to model internals such as fused layers, MoE routing, offset norms, or special attention layouts.
9. Transformers integration and data utilities
transformers/ is the Hugging Face-facing integration layer. It includes dataset loaders, compression helpers, tracing/debug utilities, and sparse/compressed-tensor helpers. This layer is the bridge between generic PyTorch/HF models and the compression engine.
10. PyTorch and infrastructure utilities
pytorch/ and utils/ provide lower-level helpers for module inspection, sparsification info, distributed utilities, metric logging, and general PyTorch/Transformers glue. These are not algorithms themselves; they support model traversal, device handling, and distributed execution.
11. Big-model and distributed support
The README and examples show explicit support for large-model loading, disk offloading, sequential onloading, and distributed calibration/quantization. The codebase includes DDP examples and utilities for model dispatch/offload, with some offloading functionality delegated to compressed-tensors. This is an important boundary for scaling to models that do not fit in memory.
12. Serialization and deployment target
The end product is a compressed checkpoint saved in a format that vLLM can consume directly. The repository is therefore not just a training-time library; it is a model transformation and export system whose downstream runtime target is vLLM.
13. Examples as architecture coverage
The examples/ tree is extensive and mirrors the supported feature set: W4A16, W8A8, FP8, NVFP4, MXFP4/MXFP8, KV-cache quantization, attention quantization, MoE, multimodal vision/audio, model-free PTQ, and large-model offloading. These examples act as executable documentation for the supported flows and model families.
14. Tests and quality gates
The test suite is broad and maps closely to the architecture: unit tests for core state/events, algorithm tests for modifiers/observers/pipelines, transformer integration tests, sparsity tests, and end-to-end vLLM validation. CI workflows in .github/workflows/ enforce these boundaries with separate checks for examples, transformer integration, and device-specific test runs.
Overall, the architecture is a layered compression framework: configuration and recipes at the top, pipeline/session orchestration in the middle, algorithm-specific modifiers/observers/model patches below that, and PyTorch/Hugging Face/vLLM integration at the edges for loading, calibration, export, and validation.
Main entry points: oneshot.py One-shot API · interface.py Modifier API · pipeline.py Sequential pipe · session.py Session
Components 19
Entry points
Public APIs for one-shot and model-free compression runs.
- One-shot API · Standard user-facing compression entrypoint.
src/llmcompressor/entrypoints/oneshot.py - Model-free PTQ · Alternative pathway for model-free post-training quantization.
src/llmcompressor/entrypoints/model_free/process.py
Orchestration
Session, lifecycle, and pipeline control plane.
- Recipe · Compression configuration and metadata abstraction.
src/llmcompressor/recipe/recipe.py - Session · Compression session state and coordination.
src/llmcompressor/core/session.py - Lifecycle · Sequencing hooks and execution lifecycle for compression steps.
src/llmcompressor/core/lifecycle.py - Sequential pipe · Ordered execution strategy for calibration and modifier application.
src/llmcompressor/pipelines/sequential/pipeline.py - Data-free pipe · Execution strategy for calibration without standard data passes.
src/llmcompressor/pipelines/data_free/pipeline.py
Algorithms
Recipe-driven modifiers, observers, and technique families.
- Modifier API · Common contract implemented by compression behaviors.
src/llmcompressor/modifiers/interface.py - Quant base · Base quantization behavior and shared mixins.
src/llmcompressor/modifiers/quantization/quantization/base.py - GPTQ · GPTQ-specific quantization modifier logic.
src/llmcompressor/modifiers/gptq/gptq_quantize.py - SparseGPT · Structured sparsification workflow for pruning-heavy compression.
src/llmcompressor/modifiers/pruning/sparsegpt/sgpt_sparsify.py - AWQ transform · Model rewrite and transformation path used by AWQ-style flows.
src/llmcompressor/modifiers/transform/awq/base.py - Observers · Calibration statistics collectors used by multiple algorithms.
src/llmcompressor/observers/base.py
Model support
Architecture-specific patches and fusion logic for target model families.
Integration
Hugging Face, PyTorch, serialization, and distributed support.
- Tensor export · Compressed-tensor and serialization helpers for downstream runtime use.
src/llmcompressor/transformers/compression/compressed_tensors_utils.py - Model load · Helpers for loading, dispatching, and offloading large models.
src/llmcompressor/pytorch/model_load/helpers.py - Distributed · Distributed execution and coordination utilities.
src/llmcompressor/utils/dist.py
Validation
Examples and tests that exercise supported compression flows.
- vLLM e2e · End-to-end validation of compressed checkpoints against vLLM.
tests/e2e/vLLM/test_vllm.py
Connections 21
- One-shot API → Recipe: loads · One-shot compression begins from a recipe definition.
- One-shot API → Session: starts · Creates and drives a compression session.
- One-shot API → Sequential pipe: runs · Dispatches work through a pipeline strategy.
- Model-free PTQ → Lifecycle: orchestrates · Model-free PTQ uses its own lifecycle and process management.
- Session → Lifecycle: coordinates · Session state advances through lifecycle events.
- Session → Modifier API: applies · Session drives modifier execution against the model.
- Recipe → Modifier API: describes · Recipes declare which modifiers and behaviors to use.
- Sequential pipe → Modifier API: orders · Sequential pipeline schedules modifier application.
- Sequential pipe → Observers: collects · Sequential execution relies on calibration observers.
- Data-free pipe → Observers: measures · Data-free calibration still depends on statistics collection.
- Quant base → Observers: uses · Quantization decisions are driven by observed statistics.
- GPTQ → Quant base: extends · GPTQ builds on the shared quantization base.
- SparseGPT → Modifier API: implements · SparseGPT is a modifier family in the common contract.
- AWQ transform → Modifier API: implements · AWQ transforms plug into the modifier interface.
- Llama 4 → AWQ transform: adapts · Model-specific patches feed transform-based compression.
- Qwen3 MoE → AWQ transform: adapts · MoE support specializes the generic transform path.
- Model load → One-shot API: loads models · Large-model loading is part of the compression entry flow.
- Distributed → Sequential pipe: supports · Distributed execution underpins sequential large-model runs.
- Tensor export → vLLM e2e: targets · Export format is validated by vLLM end-to-end tests.
- Model-free PTQ → Tensor export: saves · Model-free PTQ ends by writing compressed checkpoints.
- One-shot API → vLLM e2e: validated by · The primary API is exercised through vLLM integration tests.