# mit-han-lab/deepcompressor architecture

> Architecture diagram of the GitHub repository mit-han-lab/deepcompressor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mit-han-lab/deepcompressor
- Repository: https://github.com/mit-han-lab/deepcompressor

## Overview

##### Main Components
- Core Compression & Quantization Modules:
– The deepcompressor package contains separate modules for diffusion and LLM models. Under “deepcompressor/app/”, you have subdirectories for “diffusion” and “llm” – each with their own configuration, dataset handling, evaluation, neural network (nn) modifications, quantization routines, and pipeline scripts.
– The “deepcompressor/quantizer” directory holds additional quantizer configuration and implementation details.
- Backend Inference Systems:
– Under “deepcompressor/backend/”, you have implementations for multiple inference engines/systems: “nunchaku” (for diffusion), “qserve” (for LLM), and “tinychat” (another LLM system). These components are crucial for converting the offline quantization results into optimized, efficient runtime serving.
- Calibration & Utilities:
– The “deepcompressor/calib” directory includes calibration tools (configuration for lowrank, range, reorder, rotation, etc.) that prepare models for quantization.
– Other directories like “deepcompressor/csrc”, “deepcompressor/data”, and “deepcompressor/utils” contain lower-level C++/CUDA acceleration code, common tools, and general configs.
- Example Configurations & Scripts:
– The “examples” folder provides concrete sample configurations, prompt files, and deployment scripts for both diffusion and LLM models. This is critical to understand how the user interacts with the system and how different quantization settings are applied.
- Assets:

##### Relationships and Interactions
- Data Flow and Configuration:
- Model Evaluation and Datasets:
– The modules depend on defined datasets (in “dataset/”) and evaluation (in “eval/”) subdirectories. Include connections that indicate “dataset input” and “evaluation outputs”.
- Backend Integration:
- Utility & Low-Level Optimizations:
– Lower-level routines in “csrc”, “data”, and “utils” support both the algorithmic pipelines and backend systems. Highlight these as supporting service layers that provide optimized kernels and common functionality.

##### Architectural Patterns and Design Principles
- Modular Design:
– The clear separation between “app”, “backend”, “calib”, and “quantizer” suggests that each quantization or compression algorithm is developed and evaluated in an isolated yet interoperable module.
- Reusability & Extensibility:
– By segregating configuration files (in examples and internal config.py scripts), datasets, evaluation metrics, and algorithm implementations, the system is designed for both internal testing and extension by external users.
- Co-design Between Algorithms and Inference Kernels:
– The coupling between the quantization algorithms (e.g., SVDQuant and QoQ) and their corresponding inference engines (Nunchaku and QServe) highlights an architectural co-design where algorithmic choices influence and are optimized with backend systems.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layer 1: Configuration
    subgraph "Layer 1: Configuration"
        EX["Example Configurations & Scripts"]:::config
    end

    %% Layer 2: Core Quantization Modules
    subgraph "Layer 2: Core Quantization Modules"
        DIFF["Diffusion Quantization Module"]:::core
        LLM["LLM Quantization Module"]:::core
        QUANT["Quantization Algorithm Implementations"]:::core
        subgraph "Assets"
            AD["Assets (Diffusion)"]:::asset
            AL["Assets (LLM)"]:::asset
        end
    end

    %% Layer 3: Evaluation and Calibration
    subgraph "Layer 3: Evaluation and Calibration"
        CAL["Calibration Tools"]:::eval
        DE["Dataset & Evaluation"]:::eval
    end

    %% Layer 4: Backend Deployment Systems
    subgraph "Layer 4: Backend Deployment Systems"
        NUN["Nunchaku Inference Engine"]:::backend
        QS["QServe Inference Engine"]:::backend
        TC["TinyChat Inference Engine"]:::backend
    end

    %% Layer 5: Utilities & Low-Level Libraries
    subgraph "Layer 5: Utilities & Low-Level Libraries"
        CSRC["C++/CUDA Optimization"]:::utility
        UT["Common Utilities"]:::utility
    end

    %% Data Flow Connections
    EX -->|"configures"| DIFF
    EX -->|"configures"| LLM
    EX -->|"configures"| QUANT

    CAL -->|"preps"| DIFF
    CAL -->|"preps"| LLM

    DE -->|"feeds"| DIFF
    DE -->|"feeds"| LLM

    QUANT -->|"supports"| DIFF
    QUANT -->|"supports"| LLM

    DIFF -->|"integrates_into"| NUN
    LLM -->|"integrates_into"| QS
    LLM -->|"integrates_into"| TC

    CSRC -->|"optimizes"| DIFF
    CSRC -->|"optimizes"| LLM
    CSRC -->|"optimizes"| NUN
    CSRC -->|"optimizes"| QS
    CSRC -->|"optimizes"| TC

    UT -->|"assists"| DIFF
    UT -->|"assists"| LLM
    UT -->|"assists"| NUN
    UT -->|"assists"| QS
    UT -->|"assists"| TC

    DIFF ---|"documents"| AD
    LLM ---|"documents"| AL

    %% Click Event Mappings
    click DIFF "https://github.com/mit-han-lab/deepcompressor/tree/main/deepcompressor/app/diffusion"
    click LLM "https://github.com/mit-han-lab/deepcompressor/tree/main/deepcompressor/app/llm"
    click QUANT "https://github.com/mit-han-lab/deepcompressor/tree/main/deepcompressor/quantizer"
    click NUN "https://github.com/mit-han-lab/deepcompressor/tree/main/deepcompressor/backend/nunchaku"
    click QS "https://github.com/mit-han-lab/deepcompressor/tree/main/deepcompressor/backend/qserve"
    click TC "https://github.com/mit-han-lab/deepcompressor/tree/main/deepcompressor/backend/tinychat"
    click CAL "https://github.com/mit-han-lab/deepcompressor/tree/main/deepcompressor/calib"
    click CSRC "https://github.com/mit-han-lab/deepcompressor/tree/main/deepcompressor/csrc"
    click UT "https://github.com/mit-han-lab/deepcompressor/tree/main/deepcompressor/utils"
    click EX "https://github.com/mit-han-lab/deepcompressor/tree/main/examples"
    click AD "https://github.com/mit-han-lab/deepcompressor/tree/main/assets/diffusion"
    click AL "https://github.com/mit-han-lab/deepcompressor/tree/main/assets/llm"

    %% Style Definitions
    classDef config fill:#ffdd57,stroke:#333,stroke-width:2px;
    classDef core fill:#6ba5ff,stroke:#333,stroke-width:2px;
    classDef eval fill:#d6a8ff,stroke:#333,stroke-width:2px;
    classDef backend fill:#b6e2a1,stroke:#333,stroke-width:2px;
    classDef utility fill:#cccccc,stroke:#333,stroke-width:2px;
    classDef asset fill:#ffa3a3,stroke:#333,stroke-width:2px;
```
