# vainf/tinyfusion architecture

> Architecture diagram of the GitHub repository vainf/tinyfusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vainf/tinyfusion
- Repository: https://github.com/vainf/tinyfusion

## Overview

##### Main Components
- Diffusion Module: This is represented by the “diffusion” directory, which contains the core functions such as Gaussian diffusion, timestep sampling, utilities, and any re-spacing functions that modify the behavior of the diffusion process.
- Model Definitions: Shown in “models.py” and “models_with_layer_pruning.py”, these files define the diffusion transformer architectures and incorporate layer pruning mechanisms.
- Pruning Strategies: Several scripts (prune_by_learning.py, prune_by_score.py, prune_by_index.py) implement different pruning algorithms:
- Learnable pruning (optimization-based pruning with gradient updates and LoRA adaptations).
- Pruning by BI Score (estimating input/output similarity).
- Oracle or predefined index-based pruning.
- Training & Fine-tuning: Represented by “train.py” and “train_masked_kd.py” which implement the training pipelines for both standard fine-tuning and Masked Knowledge Distillation. These are the modules that enforce the recoverability and performance improvement on the shallow networks.
- Feature Extraction: “extract_features.py” is used to extract and encode ImageNet features, a preprocessing step to speed up training.
- Sampling & Evaluation: “sample.py” and “sample_ddp.py” are used for generating images from the pruned models whereas “evaluator.py” is geared towards computing evaluation metrics (e.g., FID). Additional visualization tools (visualize_activation.py, draw_decision.py) may showcase model performance and internal decision dynamics.

##### Establish Relationships & Data Flow
- Data Ingestion: Begin with the feature extraction process (using extract_features.py) and the acquisition of pre-trained models. Show the data flow from raw ImageNet images (or encoded features) to downstream training modules.
- Fine-tuning Pipeline: Branch from the pruned model to standard training (train.py) and Masked KD training (train_masked_kd.py) pipelines; these paths may incorporate different parameters such as “delta-w” or “lora” adjustments. Highlight that Masked KD includes a teacher–student setup.
- Sampling & Evaluation: Finally, show that the fine-tuned models feed into sampling scripts and evaluation tools. This creates a loop back to the assessment of performance, which ultimately informs further refinements.

##### Architectural Patterns and Design Principles
- Modular Architecture: The project is structured into distinct components (diffusion logic, model architectures, pruning strategies, training routines) that can be individually modified or extended.
- Pipeline-based Flow: The implementation follows a clear pipeline — from pre-trained model acquisition through to pruning, fine-tuning, and sampling. Emphasize the sequential stages and the dependencies between modules.
- Experimental Flexibility: Multiple pruning approaches and training strategies (including knowledge distillation with masked activations) indicate a design that supports extensive experimentation.

##### Relevant Technologies and Frameworks
- PyTorch is implied by the use of “torchrun” commands and the training/evaluation scripts.
- LoRA (Low-Rank Adaptation) is specifically referenced in the training routines.
- External frameworks such as guided-diffusion (from OpenAI) and MaskLLM (for Gumbel softmax) suggest dependencies for evaluation and additional functionalities.
- That the use of pre-trained weights and ImageNet as the dataset are key aspects of the implementation.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data & Feature Extraction
    subgraph "Data & Feature Extraction"
        FE("Extract ImageNet Features"):::extraction
    end

    %% Model & Diffusion Setup
    subgraph "Model & Diffusion Setup"
        DC("Diffusion Core"):::module
        M1("Base Model Definitions"):::module
        M2("Layer Pruning Models"):::module
        PM("Pre-trained Models"):::data
    end

    %% Pruning Strategies
    subgraph "Pruning Strategies"
        LP("Learnable Pruning"):::strategy
        BP("BIScore Pruning"):::strategy
        IP("Index-based Pruning"):::strategy
        PM2("Pruned Model"):::data
    end

    %% Training & Fine-tuning
    subgraph "Training & Fine-tuning"
        ST("Standard Fine-tuning Training"):::training
        MKD("Masked KD Training"):::training
        TM("Trained Model"):::data
    end

    %% Sampling & Evaluation
    subgraph "Sampling & Evaluation"
        S1("Image Sampling"):::sampling
        S2("Distributed Sampling"):::sampling
        EV1("Evaluation"):::evaluation
        VV1("Decision Visualization"):::evaluation
        VV2("Activation Visualization"):::evaluation
    end

    %% Flow Connections
    FE --> PM
    DC --> PM
    M1 --> PM
    M2 --> PM

    PM --> LP
    PM --> BP
    PM --> IP

    LP --> PM2
    BP --> PM2
    IP --> PM2

    PM2 --> ST
    PM2 --> MKD

    ST --> TM
    MKD --> TM

    TM --> S1
    TM --> S2
    TM --> EV1
    TM --> VV1
    TM --> VV2

    %% Teacher-Student Relationship for Masked KD
    PM ---|"teacher"| MKD

    %% Click Events
    click FE "https://github.com/vainf/tinyfusion/blob/main/extract_features.py"
    click DC "https://github.com/vainf/tinyfusion/tree/main/diffusion/"
    click M1 "https://github.com/vainf/tinyfusion/blob/main/models.py"
    click M2 "https://github.com/vainf/tinyfusion/blob/main/models_with_layer_pruning.py"
    click LP "https://github.com/vainf/tinyfusion/blob/main/prune_by_learning.py"
    click BP "https://github.com/vainf/tinyfusion/blob/main/prune_by_score.py"
    click IP "https://github.com/vainf/tinyfusion/blob/main/prune_by_index.py"
    click ST "https://github.com/vainf/tinyfusion/blob/main/train.py"
    click MKD "https://github.com/vainf/tinyfusion/blob/main/train_masked_kd.py"
    click S1 "https://github.com/vainf/tinyfusion/blob/main/sample.py"
    click S2 "https://github.com/vainf/tinyfusion/blob/main/sample_ddp.py"
    click EV1 "https://github.com/vainf/tinyfusion/blob/main/evaluator.py"
    click VV1 "https://github.com/vainf/tinyfusion/blob/main/draw_decision.py"
    click VV2 "https://github.com/vainf/tinyfusion/blob/main/visualize_activation.py"

    %% Styles
    classDef extraction fill:#a2d9ce,stroke:#1e8449,stroke-width:2px;
    classDef module fill:#aed6f1,stroke:#2471a3,stroke-width:2px;
    classDef data fill:#f9e79f,stroke:#b7950b,stroke-width:2px;
    classDef strategy fill:#fad7a0,stroke:#d35400,stroke-width:2px;
    classDef training fill:#d2b4de,stroke:#7d3c98,stroke-width:2px;
    classDef sampling fill:#a9dfbf,stroke:#27ae60,stroke-width:2px;
    classDef evaluation fill:#f5cba7,stroke:#d35400,stroke-width:2px;
```
