# vllm-project/llm-compressor architecture

> Architecture diagram of the GitHub repository vllm-project/llm-compressor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vllm-project/llm-compressor
- Repository: https://github.com/vllm-project/llm-compressor

## Overview

LLM Compressor is a Python library for post-training model compression, mainly for Hugging Face transformer models, with outputs intended to run in vLLM. The repository is organized around a small core engine plus algorithm-specific modifiers, model-specific support code, and many end-to-end examples/tests.

1. **Top-level runtime flow**
   The main user-facing path is: load a model → choose a compression recipe/modifier → run an entrypoint like `oneshot` or `model_free_ptq` → apply calibration/analysis/rewrites → save a compressed checkpoint in a vLLM-compatible `safetensors`-based format.

2. **Public entrypoints**
   `src/llmcompressor/entrypoints/oneshot.py` is the standard one-shot compression API.  
   `src/llmcompressor/entrypoints/model_free/` is a separate pathway for model-free PTQ, used when Hugging Face model definitions are unavailable or unreliable. It has its own lifecycle, validation, reindexing, save utilities, and process orchestration.

3. **Recipe and modifier abstraction**
   Compression is driven by `recipe` objects and `Modifier` implementations. The `recipe` package defines the configuration/metadata layer, while `modifiers/` contains the actual algorithm behaviors. This is the central extensibility boundary: recipes describe what to do; modifiers implement how to do it.

4. **Lifecycle and state management**
   The `core/` package provides session/state/event/lifecycle coordination. This is the control plane that sequences compression steps, tracks model state, and emits events across the pipeline. It keeps orchestration separate from algorithm code.

5. **Compression algorithm families**
   The modifiers are grouped by technique:
   - `quantization/` for PTQ-style quantization behaviors
   - `gptq/`, `awq/`, `autoround/`, `smoothquant/`, `obcq/`
   - `pruning/` for sparsity-related workflows
   - `transform/` for model rewrites and layout changes such as AWQ, SpinQuant, QuIP, iMatrix, SmoothQuant transforms
   - `experimental/` modifiers for newer or less stable paths
   This structure reflects a plugin-like architecture where algorithm families share common interfaces but have specialized logic.

6. **Observers and calibration**
   `observers/` contains measurement components such as min/max, MSE, moving statistics, and iMatrix observers. These collect activation or weight statistics during calibration passes and feed quantization decisions. This is the measurement layer used by multiple compression schemes.

7. **Pipelines as execution strategies**
   `pipelines/` contains the orchestration strategies: basic, data-free, independent, and sequential. These determine how modifiers are run, in what order, and how model passes are coordinated. The sequential pipeline includes AST helpers because it needs to analyze or wrap model execution paths for calibration/inference tracing.

8. **Modeling and model-specific support**
   `modeling/` holds architecture-specific calibration/fusion/patch logic for families like Llama 4, Qwen3, Gemma4, DeepSeek V3/V3.2, Granite4, GLM MoE, and Qwen multimodal/MoE variants. This is where the repo adapts generic compression logic to model internals such as fused layers, MoE routing, offset norms, or special attention layouts.

9. **Transformers integration and data utilities**
   `transformers/` is the Hugging Face-facing integration layer. It includes dataset loaders, compression helpers, tracing/debug utilities, and sparse/compressed-tensor helpers. This layer is the bridge between generic PyTorch/HF models and the compression engine.

10. **PyTorch and infrastructure utilities**
   `pytorch/` and `utils/` provide lower-level helpers for module inspection, sparsification info, distributed utilities, metric logging, and general PyTorch/Transformers glue. These are not algorithms themselves; they support model traversal, device handling, and distributed execution.

11. **Big-model and distributed support**
   The README and examples show explicit support for large-model loading, disk offloading, sequential onloading, and distributed calibration/quantization. The codebase includes DDP examples and utilities for model dispatch/offload, with some offloading functionality delegated to `compressed-tensors`. This is an important boundary for scaling to models that do not fit in memory.

12. **Serialization and deployment target**
   The end product is a compressed checkpoint saved in a format that vLLM can consume directly. The repository is therefore not just a training-time library; it is a model transformation and export system whose downstream runtime target is vLLM.

13. **Examples as architecture coverage**
   The `examples/` tree is extensive and mirrors the supported feature set: W4A16, W8A8, FP8, NVFP4, MXFP4/MXFP8, KV-cache quantization, attention quantization, MoE, multimodal vision/audio, model-free PTQ, and large-model offloading. These examples act as executable documentation for the supported flows and model families.

14. **Tests and quality gates**
   The test suite is broad and maps closely to the architecture: unit tests for core state/events, algorithm tests for modifiers/observers/pipelines, transformer integration tests, sparsity tests, and end-to-end vLLM validation. CI workflows in `.github/workflows/` enforce these boundaries with separate checks for examples, transformer integration, and device-specific test runs.

Overall, the architecture is a layered compression framework: configuration and recipes at the top, pipeline/session orchestration in the middle, algorithm-specific modifiers/observers/model patches below that, and PyTorch/Hugging Face/vLLM integration at the edges for loading, calibration, export, and validation.

Main entry points: [`src/llmcompressor/entrypoints/oneshot.py`](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/entrypoints/oneshot.py) (One-shot API), [`src/llmcompressor/modifiers/interface.py`](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/interface.py) (Modifier API), [`src/llmcompressor/pipelines/sequential/pipeline.py`](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/pipelines/sequential/pipeline.py) (Sequential pipe), [`src/llmcompressor/core/session.py`](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/core/session.py) (Session)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entrypoints["Entry points"]
  node_src_llmcompressor_entrypoints_oneshot_py["One-shot API<br/>entrypoint<br/>[oneshot.py]"]
  node_src_llmcompressor_entrypoints_model_free_process_py["Model-free PTQ<br/>entrypoint<br/>[process.py]"]
end

subgraph group_orchestration["Orchestration"]
  node_src_llmcompressor_recipe_recipe_py["Recipe<br/>config layer<br/>[recipe.py]"]
  node_src_llmcompressor_core_session_py(("Session<br/>state machine<br/>[session.py]"))
  node_src_llmcompressor_core_lifecycle_py(("Lifecycle<br/>control plane<br/>[lifecycle.py]"))
  node_src_llmcompressor_pipelines_sequential_pipeline_py{{"Sequential pipe<br/>pipeline<br/>[pipeline.py]"}}
  node_src_llmcompressor_pipelines_data_free_pipeline_py{{"Data-free pipe<br/>pipeline<br/>[pipeline.py]"}}
end

subgraph group_algorithms["Algorithms"]
  node_src_llmcompressor_modifiers_interface_py["Modifier API<br/>extension point<br/>[interface.py]"]
  node_src_llmcompressor_modifiers_quantization_quantization_base_py["Quant base<br/>quantization core<br/>[base.py]"]
  node_src_llmcompressor_modifiers_gptq_gptq_quantize_py["GPTQ<br/>quantization family<br/>[gptq_quantize.py]"]
  node_src_llmcompressor_modifiers_pruning_sparsegpt_sgpt_sparsify_py["SparseGPT<br/>sparsity family<br/>[sgpt_sparsify.py]"]
  node_src_llmcompressor_modifiers_transform_awq_base_py["AWQ transform<br/>rewrite family<br/>[base.py]"]
  node_src_llmcompressor_observers_base_py(("Observers<br/>measurement layer<br/>[base.py]"))
end

subgraph group_modeling["Model support"]
  node_src_llmcompressor_modeling_llama4_py["Llama 4<br/>model patch<br/>[llama4.py]"]
  node_src_llmcompressor_modeling_qwen3_moe_py["Qwen3 MoE<br/>model patch<br/>[qwen3_moe.py]"]
end

subgraph group_integration["Integration"]
  node_src_llmcompressor_transformers_compression_compressed_tensors_utils_py["Tensor export<br/>hf integration"]
  node_src_llmcompressor_pytorch_model_load_helpers_py["Model load<br/>pytorch glue<br/>[helpers.py]"]
  node_src_llmcompressor_utils_dist_py(("Distributed<br/>runtime support<br/>[dist.py]"))
end

subgraph group_validation["Validation"]
  node_tests_e2e_vllm_test_vllm_py["vLLM e2e<br/>validation<br/>[test_vllm.py]"]
end

node_src_llmcompressor_entrypoints_oneshot_py -->|"loads"| node_src_llmcompressor_recipe_recipe_py
node_src_llmcompressor_entrypoints_oneshot_py -->|"starts"| node_src_llmcompressor_core_session_py
node_src_llmcompressor_entrypoints_oneshot_py -->|"runs"| node_src_llmcompressor_pipelines_sequential_pipeline_py
node_src_llmcompressor_entrypoints_model_free_process_py -->|"orchestrates"| node_src_llmcompressor_core_lifecycle_py
node_src_llmcompressor_core_session_py -->|"coordinates"| node_src_llmcompressor_core_lifecycle_py
node_src_llmcompressor_core_session_py -->|"applies"| node_src_llmcompressor_modifiers_interface_py
node_src_llmcompressor_recipe_recipe_py -->|"describes"| node_src_llmcompressor_modifiers_interface_py
node_src_llmcompressor_pipelines_sequential_pipeline_py -->|"orders"| node_src_llmcompressor_modifiers_interface_py
node_src_llmcompressor_pipelines_sequential_pipeline_py -->|"collects"| node_src_llmcompressor_observers_base_py
node_src_llmcompressor_pipelines_data_free_pipeline_py -->|"measures"| node_src_llmcompressor_observers_base_py
node_src_llmcompressor_modifiers_quantization_quantization_base_py -->|"uses"| node_src_llmcompressor_observers_base_py
node_src_llmcompressor_modifiers_gptq_gptq_quantize_py -->|"extends"| node_src_llmcompressor_modifiers_quantization_quantization_base_py
node_src_llmcompressor_modifiers_pruning_sparsegpt_sgpt_sparsify_py -->|"implements"| node_src_llmcompressor_modifiers_interface_py
node_src_llmcompressor_modifiers_transform_awq_base_py -->|"implements"| node_src_llmcompressor_modifiers_interface_py
node_src_llmcompressor_modeling_llama4_py -->|"adapts"| node_src_llmcompressor_modifiers_transform_awq_base_py
node_src_llmcompressor_modeling_qwen3_moe_py -->|"adapts"| node_src_llmcompressor_modifiers_transform_awq_base_py
node_src_llmcompressor_pytorch_model_load_helpers_py -->|"loads models"| node_src_llmcompressor_entrypoints_oneshot_py
node_src_llmcompressor_utils_dist_py -->|"supports"| node_src_llmcompressor_pipelines_sequential_pipeline_py
node_src_llmcompressor_transformers_compression_compressed_tensors_utils_py -->|"targets"| node_tests_e2e_vllm_test_vllm_py
node_src_llmcompressor_entrypoints_model_free_process_py -->|"saves"| node_src_llmcompressor_transformers_compression_compressed_tensors_utils_py
node_src_llmcompressor_entrypoints_oneshot_py -->|"validated by"| node_tests_e2e_vllm_test_vllm_py

click node_src_llmcompressor_entrypoints_oneshot_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/entrypoints/oneshot.py"
click node_src_llmcompressor_entrypoints_model_free_process_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/entrypoints/model_free/process.py"
click node_src_llmcompressor_recipe_recipe_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/recipe/recipe.py"
click node_src_llmcompressor_core_session_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/core/session.py"
click node_src_llmcompressor_core_lifecycle_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/core/lifecycle.py"
click node_src_llmcompressor_pipelines_sequential_pipeline_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/pipelines/sequential/pipeline.py"
click node_src_llmcompressor_pipelines_data_free_pipeline_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/pipelines/data_free/pipeline.py"
click node_src_llmcompressor_modifiers_interface_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/interface.py"
click node_src_llmcompressor_modifiers_quantization_quantization_base_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/quantization/quantization/base.py"
click node_src_llmcompressor_modifiers_gptq_gptq_quantize_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/gptq/gptq_quantize.py"
click node_src_llmcompressor_modifiers_pruning_sparsegpt_sgpt_sparsify_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/pruning/sparsegpt/sgpt_sparsify.py"
click node_src_llmcompressor_modifiers_transform_awq_base_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/transform/awq/base.py"
click node_src_llmcompressor_observers_base_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/observers/base.py"
click node_src_llmcompressor_modeling_llama4_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modeling/llama4.py"
click node_src_llmcompressor_modeling_qwen3_moe_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modeling/qwen3_moe.py"
click node_src_llmcompressor_transformers_compression_compressed_tensors_utils_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/transformers/compression/compressed_tensors_utils.py"
click node_src_llmcompressor_pytorch_model_load_helpers_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/pytorch/model_load/helpers.py"
click node_src_llmcompressor_utils_dist_py "https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/utils/dist.py"
click node_tests_e2e_vllm_test_vllm_py "https://github.com/vllm-project/llm-compressor/blob/main/tests/e2e/vLLM/test_vllm.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_src_llmcompressor_entrypoints_oneshot_py,node_src_llmcompressor_entrypoints_model_free_process_py toneBlue
class node_src_llmcompressor_recipe_recipe_py,node_src_llmcompressor_core_session_py,node_src_llmcompressor_core_lifecycle_py,node_src_llmcompressor_pipelines_sequential_pipeline_py,node_src_llmcompressor_pipelines_data_free_pipeline_py toneAmber
class node_src_llmcompressor_modifiers_interface_py,node_src_llmcompressor_modifiers_quantization_quantization_base_py,node_src_llmcompressor_modifiers_gptq_gptq_quantize_py,node_src_llmcompressor_modifiers_pruning_sparsegpt_sgpt_sparsify_py,node_src_llmcompressor_modifiers_transform_awq_base_py,node_src_llmcompressor_observers_base_py toneMint
class node_src_llmcompressor_modeling_llama4_py,node_src_llmcompressor_modeling_qwen3_moe_py toneRose
class node_src_llmcompressor_transformers_compression_compressed_tensors_utils_py,node_src_llmcompressor_pytorch_model_load_helpers_py,node_src_llmcompressor_utils_dist_py toneIndigo
class node_tests_e2e_vllm_test_vllm_py toneTeal
```

## Components

### Entry points

Public APIs for one-shot and model-free compression runs.

- [One-shot API](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/entrypoints/oneshot.py): entrypoint · Standard user-facing compression entrypoint. · `src/llmcompressor/entrypoints/oneshot.py`
- [Model-free PTQ](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/entrypoints/model_free/process.py): entrypoint · Alternative pathway for model-free post-training quantization. · `src/llmcompressor/entrypoints/model_free/process.py`

### Orchestration

Session, lifecycle, and pipeline control plane.

- [Recipe](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/recipe/recipe.py): config layer · Compression configuration and metadata abstraction. · `src/llmcompressor/recipe/recipe.py`
- [Session](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/core/session.py): state machine · Compression session state and coordination. · `src/llmcompressor/core/session.py`
- [Lifecycle](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/core/lifecycle.py): control plane · Sequencing hooks and execution lifecycle for compression steps. · `src/llmcompressor/core/lifecycle.py`
- [Sequential pipe](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/pipelines/sequential/pipeline.py): pipeline · Ordered execution strategy for calibration and modifier application. · `src/llmcompressor/pipelines/sequential/pipeline.py`
- [Data-free pipe](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/pipelines/data_free/pipeline.py): pipeline · Execution strategy for calibration without standard data passes. · `src/llmcompressor/pipelines/data_free/pipeline.py`

### Algorithms

Recipe-driven modifiers, observers, and technique families.

- [Modifier API](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/interface.py): extension point · Common contract implemented by compression behaviors. · `src/llmcompressor/modifiers/interface.py`
- [Quant base](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/quantization/quantization/base.py): quantization core · Base quantization behavior and shared mixins. · `src/llmcompressor/modifiers/quantization/quantization/base.py`
- [GPTQ](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/gptq/gptq_quantize.py): quantization family · GPTQ-specific quantization modifier logic. · `src/llmcompressor/modifiers/gptq/gptq_quantize.py`
- [SparseGPT](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/pruning/sparsegpt/sgpt_sparsify.py): sparsity family · Structured sparsification workflow for pruning-heavy compression. · `src/llmcompressor/modifiers/pruning/sparsegpt/sgpt_sparsify.py`
- [AWQ transform](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/transform/awq/base.py): rewrite family · Model rewrite and transformation path used by AWQ-style flows. · `src/llmcompressor/modifiers/transform/awq/base.py`
- [Observers](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/observers/base.py): measurement layer · Calibration statistics collectors used by multiple algorithms. · `src/llmcompressor/observers/base.py`

### Model support

Architecture-specific patches and fusion logic for target model families.

- [Llama 4](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modeling/llama4.py): model patch · Model-specific calibration and fusion support. · `src/llmcompressor/modeling/llama4.py`
- [Qwen3 MoE](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modeling/qwen3_moe.py): model patch · MoE-specific support and adaptation logic. · `src/llmcompressor/modeling/qwen3_moe.py`

### Integration

Hugging Face, PyTorch, serialization, and distributed support.

- [Tensor export](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/transformers/compression/compressed_tensors_utils.py): hf integration · Compressed-tensor and serialization helpers for downstream runtime use. · `src/llmcompressor/transformers/compression/compressed_tensors_utils.py`
- [Model load](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/pytorch/model_load/helpers.py): pytorch glue · Helpers for loading, dispatching, and offloading large models. · `src/llmcompressor/pytorch/model_load/helpers.py`
- [Distributed](https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/utils/dist.py): runtime support · Distributed execution and coordination utilities. · `src/llmcompressor/utils/dist.py`

### Validation

Examples and tests that exercise supported compression flows.

- [vLLM e2e](https://github.com/vllm-project/llm-compressor/blob/main/tests/e2e/vLLM/test_vllm.py): validation · End-to-end validation of compressed checkpoints against vLLM. · `tests/e2e/vLLM/test_vllm.py`

## Connections

- One-shot API → Recipe: loads · One-shot compression begins from a recipe definition.
- One-shot API → Session: starts · Creates and drives a compression session.
- One-shot API → Sequential pipe: runs · Dispatches work through a pipeline strategy.
- Model-free PTQ → Lifecycle: orchestrates · Model-free PTQ uses its own lifecycle and process management.
- Session → Lifecycle: coordinates · Session state advances through lifecycle events.
- Session → Modifier API: applies · Session drives modifier execution against the model.
- Recipe → Modifier API: describes · Recipes declare which modifiers and behaviors to use.
- Sequential pipe → Modifier API: orders · Sequential pipeline schedules modifier application.
- Sequential pipe → Observers: collects · Sequential execution relies on calibration observers.
- Data-free pipe → Observers: measures · Data-free calibration still depends on statistics collection.
- Quant base → Observers: uses · Quantization decisions are driven by observed statistics.
- GPTQ → Quant base: extends · GPTQ builds on the shared quantization base.
- SparseGPT → Modifier API: implements · SparseGPT is a modifier family in the common contract.
- AWQ transform → Modifier API: implements · AWQ transforms plug into the modifier interface.
- Llama 4 → AWQ transform: adapts · Model-specific patches feed transform-based compression.
- Qwen3 MoE → AWQ transform: adapts · MoE support specializes the generic transform path.
- Model load → One-shot API: loads models · Large-model loading is part of the compression entry flow.
- Distributed → Sequential pipe: supports · Distributed execution underpins sequential large-model runs.
- Tensor export → vLLM e2e: targets · Export format is validated by vLLM end-to-end tests.
- Model-free PTQ → Tensor export: saves · Model-free PTQ ends by writing compressed checkpoints.
- One-shot API → vLLM e2e: validated by · The primary API is exercised through vLLM integration tests.
