How clin0212/hydralora works
Project type and purpose
• This is a research/software framework for efficient parameter-efficient fine-tuning of large language models (LLMs), focused on Asymmetric LoRA (“HydraLoRA”) with an MoE router for discovery and fusion of intrinsic components. It also contains an extension (“MLLM-HydraLoRA”) for multimodal LLMs (Llava) and a small “Motivation” module for analysis of LoRA submodules.
File structure
• Top-level directories
• HydraLoRA – core implementation: data preparation, fine-tuning scripts, a fork of PEFT modules, a fork of Transformers
• MLLM-HydraLoRA – multimodal adaptation builder, patches for MoE, train script, wrapper for Llava/LLaMA, export utilities
• Motivation – scripts and shell wrappers for analyzing LoRA modules
• Key scripts
• build_dataset.py – loads JSON examples, builds PyTorch datasets/data loaders
• fine-tuning.py / fine-tuning.sh / fine-tuning_dp.sh – single-GPU and DeepSpeed entrypoints
• MLLM_HydraLoRA.sh, builder.py, train.py – orchestrate multimodal model setup and training
• Configuration and dependencies
• environment.yml / requirements.txt – conda and pip dependencies (PyTorch, transformers, peft, deepspeed)
• ds_zero3_offload.json – DeepSpeed Zero3 offload config
• Custom modules
• HydraLoRA/peft and HydraLoRA/transformers – forked/extended modules to inject asymmetric LoRA logic
• MLLM-HydraLoRA/moe and moe_patch – MoE router implementation and patch for llama internals
README for additional insights
• Purpose: implement and evaluate an asymmetric LoRA architecture that shares A matrices and uses multiple B matrices routed by an MoE expert router
• Workflow:
a. N intrinsic components (e.g., via k-means)
b. Initialize shared A and N distinct B LoRA submodules
c. Train with MoE router assigning samples to experts
d. Merge B modules during inference via trained router
• Integration points: PyTorch training loop, PEFT integration, Transformers models, DeepSpeed for large models, OpenCompass for evaluation
Elements
• Rectangular nodes labeled: Data Loader, Transformer Base Model, Shared A LoRA Adapter, B Expert 1…B Expert N, MoE Router, Trainer (PyTorch/DeepSpeed), Checkpoint I/O, Evaluation Client
• Blue for data components
• Green for model and adapters
• Orange for routing/expert components
• Gray for external integration (OpenCompass)
• Rounded rectangles for processes (dataset builder, trainer)
• Stacked rectangles for multiple B experts
• Diamonds for decision/routing (MoE router)