How horseee/llm-pruner works
Project Type and Purpose
• This is a research/engineering tool that automates the structural pruning of large language models. It supports multiple LLM architectures (LLaMA, Llama-2, Llama-3, BLOOM, Vicuna, ChatGLM, Baichuan, etc.) and includes facilities for post-training (using LoRA/PEFT techniques) and evaluation (integrated with lm-evaluation-harness).
• The README indicates that the tool is used for “task-agnostic compression” of LLMs, featuring a three-stage process – Discovery, Estimation, and Recovery – and supporting model generation and evaluation.
Key Components
• Data and Dataset Management:
• A “datasets” submodule contains scripts for handling various datasets (dialogue, example_samples, ppl_dataset).
• The lm-evaluation-harness directory also includes multiple dataset definitions and pre-processing scripts.
• Model Abstraction and Wrappers:
• Under LLMPruner/models, separate folders for different model families (hf_baichuan, hf_bloom, hf_chatglm, hf_llama, and a legacy “llama” folder) represent the abstraction over different model implementations.
• These modules include configuration, tokenizer, and modeling files, encapsulating the architecture of each LLM variant.
• Structural Pruning Engine:
• The “pruner” directory provides model-specific pruning scripts (e.g., hf_baichuan_pruner.py, hf_chatglm_pruner.py, etc.) for running the discovery (dependency analysis) and estimation stages.
• In addition, the “torch_pruning” submodule contains helper utilities and a set of algorithms (magnitude based, batchnorm-scale, Taylor’s approximations, etc.) to execute the actual pruning process.
• Post-Training (Model Recovery / Fine-Tuning):
• The “peft” directory implements parameter-efficient fine-tuning techniques including various tuners (lora, adalora, prefix tuning, etc.) to recover performance after pruning.
• The post_training.py script brings together pruned-model loading, training routines, and configuration options to conduct recovery using external datasets like Alpaca or LaMini.
• Evaluation and Generation:
• The evaluator folder and scripts (e.g., hf_prune.py, test_speedup.py, generate.py) tie together the evaluation stage – including zero-shot tests, performance measurements, and a generation interface (with Gradio integration).
• The lm-evaluation-harness subdirectory (entire duplicate inclusion) shows that the project integrates external evaluation benchmarks and orchestration tasks (tasks, metrics, etc.).
• Auxiliary Tools and Utilities:
• The “utils” package under LLMPruner includes helper modules for logging, prompt management, and other miscellaneous functions.
• There’s also an “examples” folder with example scripts that demonstrate pruning for specific models.
Architectural Relationships and Data Flow
• Pruning Pipeline: The pruner components use the base model to scan for inter-dependencies and then estimate the importance of “groups” of parameters using techniques defined under “torch_pruning”. These two stages (discovery and estimation) are executed sequentially.
• Post-Training Recovery: The pruned model is fed to the PEFT-based recovery module, with training scripts (post_training.py) responsible for fine-tuning using a limited dataset.
• Evaluation & Generation: Once post-training is complete, the model is either evaluated using the integrated lm-evaluation-harness or used for text generation via generate.py. This stage typically involves separate modules to handle test datasets and generate outputs.
Architectural Patterns and Design Principles
• Modular Design: The codebase is highly modular – each major function (data handling, model abstraction, pruning, post-training, evaluation) is separated into its own package.
• Plug-in Flexibility: Different pruning strategies (block-wise, channel-wise, layer-wise) and various importance criteria (taylor, l1, l2, random) are supported through distinct modules, indicating configurable pipelines.
• Integration with External Tools: The system integrates an external evaluation harness as a sub-system, so showing a connection to an “Evaluation Module” is important.