# horseee/llm-pruner architecture

> Architecture diagram of the GitHub repository horseee/llm-pruner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/horseee/llm-pruner
- Repository: https://github.com/horseee/llm-pruner

## Overview

##### Project Type and Purpose
- This is a research/engineering tool that automates the structural pruning of large language models. It supports multiple LLM architectures (LLaMA, Llama-2, Llama-3, BLOOM, Vicuna, ChatGLM, Baichuan, etc.) and includes facilities for post-training (using LoRA/PEFT techniques) and evaluation (integrated with lm-evaluation-harness).
- The README indicates that the tool is used for “task-agnostic compression” of LLMs, featuring a three-stage process – Discovery, Estimation, and Recovery – and supporting model generation and evaluation.

##### Key Components
- Data and Dataset Management:
- A “datasets” submodule contains scripts for handling various datasets (dialogue, example_samples, ppl_dataset).
- The lm-evaluation-harness directory also includes multiple dataset definitions and pre-processing scripts.
- Model Abstraction and Wrappers:
- Under LLMPruner/models, separate folders for different model families (hf_baichuan, hf_bloom, hf_chatglm, hf_llama, and a legacy “llama” folder) represent the abstraction over different model implementations.
- These modules include configuration, tokenizer, and modeling files, encapsulating the architecture of each LLM variant.
- Structural Pruning Engine:
- The “pruner” directory provides model-specific pruning scripts (e.g., hf_baichuan_pruner.py, hf_chatglm_pruner.py, etc.) for running the discovery (dependency analysis) and estimation stages.
- In addition, the “torch_pruning” submodule contains helper utilities and a set of algorithms (magnitude based, batchnorm-scale, Taylor’s approximations, etc.) to execute the actual pruning process.
- Post-Training (Model Recovery / Fine-Tuning):
- The “peft” directory implements parameter-efficient fine-tuning techniques including various tuners (lora, adalora, prefix tuning, etc.) to recover performance after pruning.
- The post_training.py script brings together pruned-model loading, training routines, and configuration options to conduct recovery using external datasets like Alpaca or LaMini.
- Evaluation and Generation:
- The evaluator folder and scripts (e.g., hf_prune.py, test_speedup.py, generate.py) tie together the evaluation stage – including zero-shot tests, performance measurements, and a generation interface (with Gradio integration).
- The lm-evaluation-harness subdirectory (entire duplicate inclusion) shows that the project integrates external evaluation benchmarks and orchestration tasks (tasks, metrics, etc.).
- Auxiliary Tools and Utilities:
- The “utils” package under LLMPruner includes helper modules for logging, prompt management, and other miscellaneous functions.
- There’s also an “examples” folder with example scripts that demonstrate pruning for specific models.

##### Architectural Relationships and Data Flow
- Pruning Pipeline: The pruner components use the base model to scan for inter-dependencies and then estimate the importance of “groups” of parameters using techniques defined under “torch_pruning”. These two stages (discovery and estimation) are executed sequentially.
- Post-Training Recovery: The pruned model is fed to the PEFT-based recovery module, with training scripts (post_training.py) responsible for fine-tuning using a limited dataset.
- Evaluation & Generation: Once post-training is complete, the model is either evaluated using the integrated lm-evaluation-harness or used for text generation via generate.py. This stage typically involves separate modules to handle test datasets and generate outputs.

##### Architectural Patterns and Design Principles
- Modular Design: The codebase is highly modular – each major function (data handling, model abstraction, pruning, post-training, evaluation) is separated into its own package.
- Plug-in Flexibility: Different pruning strategies (block-wise, channel-wise, layer-wise) and various importance criteria (taylor, l1, l2, random) are supported through distinct modules, indicating configurable pipelines.
- Integration with External Tools: The system integrates an external evaluation harness as a sub-system, so showing a connection to an “Evaluation Module” is important.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Management Layer
    A["LLMPruner/datasets"]:::dataset
    B["lm-evaluation-harness/lm_eval/datasets"]:::dataset

    %% Model Abstraction Layer
    C["LLMPruner/models"]:::model

    %% Structural Pruning Engine Layer
    D["LLMPruner/pruner"]:::pruner
    E["LLMPruner/torch_pruning"]:::pruner

    %% Post-Training / Fine-Tuning Layer
    F["LLMPruner/peft"]:::posttraining
    G["post_training.py"]:::posttraining

    %% Evaluation & Generation Layer
    H["LLMPruner/evaluator"]:::evaluator
    I["generate.py"]:::evaluator
    J["hf_prune.py and test_speedup.py"]:::evaluator
    K["lm-evaluation-harness"]:::evaluator

    %% Auxiliary Tools and Utilities (side components)
    L["LLMPruner/utils"]:::utilities
    M["examples"]:::utilities

    %% External Dependencies (side component)
    N["External Libraries: PyTorch, Huggingface Transformers, deepspeed"]:::external

    %% Connections
    %% External Dependencies support Model Loading and Post-Training/Evaluation
    N -->|"supports"| C
    N -.->|"supports"| F
    N -.->|"supports"| H

    %% Model Loader receives support from Utilities
    L -.->|"logs/help"| C
    L -.->|"logs/help"| D
    L -.->|"logs/help"| F
    M -.->|"demo"| F

    %% Model Loading and Pruning Pipeline
    C -->|"loads model"| D
    D -->|"runs discovery/estimation"| E
    E -->|"outputs pruned model"| F

    %% Post-Training Recovery
    F -->|"fine tunes pruned model"| G
    A -->|"training data"| G

    %% Evaluation & Generation
    G -->|"evaluates model"| H
    G -->|"generates text"| I
    G -->|"performance eval"| J
    H ---|"integrates"| K
    B -->|"evaluation datasets"| K

    %% Class Definitions for coloring
    classDef dataset fill:#FFC300,stroke:#333,stroke-width:2px;
    classDef model fill:#66CCFF,stroke:#333,stroke-width:2px;
    classDef pruner fill:#FF6666,stroke:#333,stroke-width:2px;
    classDef posttraining fill:#99FF99,stroke:#333,stroke-width:2px;
    classDef evaluator fill:#CC99FF,stroke:#333,stroke-width:2px;
    classDef utilities fill:#FFCC66,stroke:#333,stroke-width:2px;
    classDef external fill:#CCCCCC,stroke:#333,stroke-width:2px;

    %% Click Events
    click A "https://github.com/horseee/llm-pruner/tree/main/LLMPruner/datasets"
    click B "https://github.com/horseee/llm-pruner/tree/main/lm-evaluation-harness/lm_eval/datasets"
    click C "https://github.com/horseee/llm-pruner/tree/main/LLMPruner/models"
    click D "https://github.com/horseee/llm-pruner/tree/main/LLMPruner/pruner"
    click E "https://github.com/horseee/llm-pruner/tree/main/LLMPruner/torch_pruning"
    click F "https://github.com/horseee/llm-pruner/tree/main/LLMPruner/peft"
    click G "https://github.com/horseee/llm-pruner/blob/main/post_training.py"
    click H "https://github.com/horseee/llm-pruner/tree/main/LLMPruner/evaluator"
    click I "https://github.com/horseee/llm-pruner/blob/main/generate.py"
    click J "https://github.com/horseee/llm-pruner/blob/main/hf_prune.py"
    click K "https://github.com/horseee/llm-pruner/tree/main/lm-evaluation-harness"
    click L "https://github.com/horseee/llm-pruner/tree/main/LLMPruner/utils"
    click M "https://github.com/horseee/llm-pruner/tree/main/examples"
```
