How vainf/tinyfusion works
Main Components
• Diffusion Module: This is represented by the “diffusion” directory, which contains the core functions such as Gaussian diffusion, timestep sampling, utilities, and any re-spacing functions that modify the behavior of the diffusion process.
• Model Definitions: Shown in “models.py” and “models_with_layer_pruning.py”, these files define the diffusion transformer architectures and incorporate layer pruning mechanisms.
• Pruning Strategies: Several scripts (prune_by_learning.py, prune_by_score.py, prune_by_index.py) implement different pruning algorithms:
• Learnable pruning (optimization-based pruning with gradient updates and LoRA adaptations).
• Pruning by BI Score (estimating input/output similarity).
• Oracle or predefined index-based pruning.
• Training & Fine-tuning: Represented by “train.py” and “train_masked_kd.py” which implement the training pipelines for both standard fine-tuning and Masked Knowledge Distillation. These are the modules that enforce the recoverability and performance improvement on the shallow networks.
• Feature Extraction: “extract_features.py” is used to extract and encode ImageNet features, a preprocessing step to speed up training.
• Sampling & Evaluation: “sample.py” and “sample_ddp.py” are used for generating images from the pruned models whereas “evaluator.py” is geared towards computing evaluation metrics (e.g., FID). Additional visualization tools (visualize_activation.py, draw_decision.py) may showcase model performance and internal decision dynamics.
Establish Relationships & Data Flow
• Data Ingestion: Begin with the feature extraction process (using extract_features.py) and the acquisition of pre-trained models. Show the data flow from raw ImageNet images (or encoded features) to downstream training modules.
• Fine-tuning Pipeline: Branch from the pruned model to standard training (train.py) and Masked KD training (train_masked_kd.py) pipelines; these paths may incorporate different parameters such as “delta-w” or “lora” adjustments. Highlight that Masked KD includes a teacher–student setup.
• Sampling & Evaluation: Finally, show that the fine-tuned models feed into sampling scripts and evaluation tools. This creates a loop back to the assessment of performance, which ultimately informs further refinements.
Architectural Patterns and Design Principles
• Modular Architecture: The project is structured into distinct components (diffusion logic, model architectures, pruning strategies, training routines) that can be individually modified or extended.
• Pipeline-based Flow: The implementation follows a clear pipeline — from pre-trained model acquisition through to pruning, fine-tuning, and sampling. Emphasize the sequential stages and the dependencies between modules.
• Experimental Flexibility: Multiple pruning approaches and training strategies (including knowledge distillation with masked activations) indicate a design that supports extensive experimentation.
Relevant Technologies and Frameworks
• PyTorch is implied by the use of “torchrun” commands and the training/evaluation scripts.
• LoRA (Low-Rank Adaptation) is specifically referenced in the training routines.
• External frameworks such as guided-diffusion (from OpenAI) and MaskLLM (for Gumbel softmax) suggest dependencies for evaluation and additional functionalities.
• That the use of pre-trained weights and ImageNet as the dataset are key aspects of the implementation.