How mit-han-lab/deepcompressor works
Main Components
• Core Compression & Quantization Modules:
– The deepcompressor package contains separate modules for diffusion and LLM models. Under “deepcompressor/app/”, you have subdirectories for “diffusion” and “llm” – each with their own configuration, dataset handling, evaluation, neural network (nn) modifications, quantization routines, and pipeline scripts.
– The “deepcompressor/quantizer” directory holds additional quantizer configuration and implementation details.
• Backend Inference Systems:
– Under “deepcompressor/backend/”, you have implementations for multiple inference engines/systems: “nunchaku” (for diffusion), “qserve” (for LLM), and “tinychat” (another LLM system). These components are crucial for converting the offline quantization results into optimized, efficient runtime serving.
• Calibration & Utilities:
– The “deepcompressor/calib” directory includes calibration tools (configuration for lowrank, range, reorder, rotation, etc.) that prepare models for quantization.
– Other directories like “deepcompressor/csrc”, “deepcompressor/data”, and “deepcompressor/utils” contain lower-level C++/CUDA acceleration code, common tools, and general configs.
• Example Configurations & Scripts:
– The “examples” folder provides concrete sample configurations, prompt files, and deployment scripts for both diffusion and LLM models. This is critical to understand how the user interacts with the system and how different quantization settings are applied.
• Assets:
Relationships and Interactions
• Data Flow and Configuration:
• Model Evaluation and Datasets:
– The modules depend on defined datasets (in “dataset/”) and evaluation (in “eval/”) subdirectories. Include connections that indicate “dataset input” and “evaluation outputs”.
• Backend Integration:
• Utility & Low-Level Optimizations:
– Lower-level routines in “csrc”, “data”, and “utils” support both the algorithmic pipelines and backend systems. Highlight these as supporting service layers that provide optimized kernels and common functionality.
Architectural Patterns and Design Principles
• Modular Design:
– The clear separation between “app”, “backend”, “calib”, and “quantizer” suggests that each quantization or compression algorithm is developed and evaluated in an isolated yet interoperable module.
• Reusability & Extensibility:
– By segregating configuration files (in examples and internal config.py scripts), datasets, evaluation metrics, and algorithm implementations, the system is designed for both internal testing and extension by external users.
• Co-design Between Algorithms and Inference Kernels:
– The coupling between the quantization algorithms (e.g., SVDQuant and QoQ) and their corresponding inference engines (Nunchaku and QServe) highlights an architectural co-design where algorithmic choices influence and are optimized with backend systems.