How modeltc/l2_compression works
Main Components
• Data Processing and Datasets
• The directory data_new provides modules for loading and preprocessing datasets (e.g., ImageNet). This is critical since the training scripts (found in the train directory) rely on these datasets.
• Model Architectures
• Two directories—cifar10_models and model—contain various network architectures (ResNet, MobileNetV2, DenseNet, etc.). These represent the selectable deep learning models on which the compression will be applied.
• Compression and Quantization Algorithms
• The compress directory holds modules for lossless compression algorithms (Huffman coding and range coding).
• The quant directory contains quantization routines that adjust weight precision (e.g., soft quant methods).
• Training and Calibration Pipelines
• The train directory includes training scripts, loggers, and experiment modules for preparing the models and running calibration. This module processes checkpoints and facilitates adjustments like weight transformations.
• The README examples point to train/train_final, which is the main entry point for adapting model checkpoints and calibrating datasets.
• Transformation Operations
• The transform directory contains various weight transformation functions (e.g., edge_scale, log, scale) that are applied during the compression process to achieve the L^2 compression objectives.
• Compiled Acceleration Libraries
• The clib directory aggregates several performance-critical routines implemented in C++ and CUDA (organized into cos_stat, lin_stat, and tri_stat subfolders). These are used to offload heavy computation tasks in functions critical for compression or statistics collection.
• Utility and Helper Functions
• The common directory contains shared code and tools that are used across modules.
• Additionally, cal_bit.py and cal_bit_huffman.py are provided as scripts for encoding/decoding and testing the compression operations.
Out Relationships and Data Flow
• Input data (e.g., ImageNet images) is loaded and preprocessed from the data_new modules.
• Model Definition and Selection:
• The appropriate model (from cifar10_models or model) is loaded and instantiated. The training scripts interact with these classes.
• Training and Calibration Pipeline:
• The train directory orchestrates the training or fine-tuning process. The trainer reads model checkpoints, configures training hyperparameters, and makes calls to both transformation and quantization modules. Calibration data (as specified in train.txt) is used to adjust parameters.
• Weight Transformation and Compression:
• The transform modules apply specific mathematical transformations (edgescale, scale, etc.) to the model weights.
• Next, the quant module further processes these weights (through quantization methods like soft quantization).
• The compress modules handle the final encoding (using Huffman or range coding) to get the compressed representation.
• Integration of Compiled Libraries:
• Specialized routines in clib (implemented in both C++ and CUDA) are invoked by higher-level Python modules for efficient computation of statistical metrics (e.g., cosine, linear, or triangular statistical measures) required during weight transformation and compression.
Architectural Patterns and Key Design Principles
• Modularity and Separation of Concerns:
• The project clearly separates concerns: model definition, data management, training, transformation, quantization, and compression are all handled by distinct modules.
• Layered Architecture:
• Extensibility:
• With the distinct directories for models and compression techniques, the design allows for replacing or extending certain components (e.g., adding new model architectures or compression algorithms) without major rework.