# clin0212/hydralora architecture

> Architecture diagram of the GitHub repository clin0212/hydralora, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/clin0212/hydralora
- Repository: https://github.com/clin0212/hydralora

## Overview

##### Project type and purpose
- This is a research/software framework for efficient parameter-efficient fine-tuning of large language models (LLMs), focused on Asymmetric LoRA (“HydraLoRA”) with an MoE router for discovery and fusion of intrinsic components. It also contains an extension (“MLLM-HydraLoRA”) for multimodal LLMs (Llava) and a small “Motivation” module for analysis of LoRA submodules.

##### File structure
- Top-level directories
- HydraLoRA – core implementation: data preparation, fine-tuning scripts, a fork of PEFT modules, a fork of Transformers
- MLLM-HydraLoRA – multimodal adaptation builder, patches for MoE, train script, wrapper for Llava/LLaMA, export utilities
- Motivation – scripts and shell wrappers for analyzing LoRA modules
- Key scripts
- build_dataset.py – loads JSON examples, builds PyTorch datasets/data loaders
- fine-tuning.py / fine-tuning.sh / fine-tuning_dp.sh – single-GPU and DeepSpeed entrypoints
- MLLM_HydraLoRA.sh, builder.py, train.py – orchestrate multimodal model setup and training
- Configuration and dependencies
- environment.yml / requirements.txt – conda and pip dependencies (PyTorch, transformers, peft, deepspeed)
- ds_zero3_offload.json – DeepSpeed Zero3 offload config
- Custom modules
- HydraLoRA/peft and HydraLoRA/transformers – forked/extended modules to inject asymmetric LoRA logic
- MLLM-HydraLoRA/moe and moe_patch – MoE router implementation and patch for llama internals

##### README for additional insights
- Purpose: implement and evaluate an asymmetric LoRA architecture that shares A matrices and uses multiple B matrices routed by an MoE expert router
- Workflow:
a. N intrinsic components (e.g., via k-means)
b. Initialize shared A and N distinct B LoRA submodules
c. Train with MoE router assigning samples to experts
d. Merge B modules during inference via trained router
- Integration points: PyTorch training loop, PEFT integration, Transformers models, DeepSpeed for large models, OpenCompass for evaluation

##### Elements
- Rectangular nodes labeled: Data Loader, Transformer Base Model, Shared A LoRA Adapter, B Expert 1…B Expert N, MoE Router, Trainer (PyTorch/DeepSpeed), Checkpoint I/O, Evaluation Client
- Blue for data components
- Green for model and adapters
- Orange for routing/expert components
- Gray for external integration (OpenCompass)
- Rounded rectangles for processes (dataset builder, trainer)
- Stacked rectangles for multiple B experts
- Diamonds for decision/routing (MoE router)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    DataSource[(Raw JSON Data)]:::data
    DataBuilder["Dataset Builder\nHydraLoRA/build_dataset.py"]:::data
    DataLoader["PyTorch DataLoader"]:::data

    %% Trainer Layer
    TrainerSG["Single-GPU Trainer\nfine-tuning.py\nfine-tuning.sh"]:::model
    DeepSpeedTrainer["DeepSpeed Trainer\nfine-tuning_dp.sh\nds_zero3_offload.json"]:::model

    %% Core Model & Adapters
    Transformers["Forked Transformers\nHydraLoRA/transformers/"]:::model
    PeftCore["Forked PEFT Core\npeft_model.py"]:::model
    SharedA["Shared A Adapter\nshared.py"]:::model
    ExpertB["B Expert Modules\nmmoelora.py, mmoeloraS.py"]:::routing
    BaseModel["Transformer Base Model"]:::model

    %% Routing
    MoERouter{"MoE Router\nMLLM-HydraLoRA/moe.py"}:::routing
    MoEPatch["MoE Patch\nmodeling_llama.py"]:::routing

    %% Checkpoint & Evaluation
    CheckpointIO["Checkpoint I/O"]:::external
    Evaluation["Evaluation Client\nOpenCompass"]:::external

    %% Multimodal Extension
    MultimodalBuilder["Multimodal Builder\nbuilder.py"]:::model
    MultiWrapper["Llava-LLaMA Wrapper\nllava_llama.py"]:::model
    MultiTrain["Multimodal Trainer\ntrain.py\nMLLM_HydraLoRA.sh"]:::model
    ExportUtils["Export Utilities\nexport/"]:::external

    %% Other Utilities
    Motivation["Motivation Analysis\ntsen_lora.py\ntesn_lora.sh"]:::external
    Figures["Figures\nfigures/"]:::external
    EnvConfig["Env & Deps\nenvironment.yml\nrequirements.txt"]:::external
    RootREADME["README.md"]:::external

    %% Data Flow
    DataSource --> DataBuilder --> DataLoader --> TrainerSG
    DataSource --> DataBuilder --> DataLoader --> DeepSpeedTrainer

    %% Training Flow
    TrainerSG --> Transformers
    TrainerSG --> PeftCore
    TrainerSG --> BaseModel
    DeepSpeedTrainer --> Transformers
    DeepSpeedTrainer --> PeftCore
    DeepSpeedTrainer --> BaseModel

    %% Adapter Injection
    BaseModel --> SharedA
    BaseModel --> ExpertB
    BaseModel --> MoERouter
    MoERouter --> ExpertB
    ExpertB --> BaseModel

    %% Patch Integration
    BaseModel --> MoEPatch

    %% Checkpoint & Eval
    TrainerSG --> CheckpointIO
    DeepSpeedTrainer --> CheckpointIO
    CheckpointIO --> ExportUtils
    Evaluation --> CheckpointIO
    Evaluation --> BaseModel

    %% Multimodal Branch
    MultimodalBuilder --> MultiWrapper --> MultiTrain --> TrainerSG
    MultiTrain --> DeepSpeedTrainer

    %% Other Utilities Connections
    Motivation -.-> BaseModel
    Figures -.-> RootREADME
    EnvConfig -.-> TrainerSG
    EnvConfig -.-> DeepSpeedTrainer
    RootREADME -.-> DataBuilder

    %% Click Events
    click DataSource "https://github.com/clin0212/hydralora/tree/main/HydraLoRA/data/"
    click DataBuilder "https://github.com/clin0212/hydralora/blob/main/HydraLoRA/build_dataset.py"
    click DataLoader "https://github.com/clin0212/hydralora/blob/main/HydraLoRA/build_dataset.py"
    click TrainerSG "https://github.com/clin0212/hydralora/blob/main/HydraLoRA/fine-tuning.py"
    click DeepSpeedTrainer "https://github.com/clin0212/hydralora/blob/main/HydraLoRA/fine-tuning_dp.sh"
    click PeftCore "https://github.com/clin0212/hydralora/blob/main/HydraLoRA/peft/peft_model.py"
    click PeftCore "https://github.com/clin0212/hydralora/blob/main/HydraLoRA/peft/peft/peft_model.py"
    click SharedA "https://github.com/clin0212/hydralora/blob/main/HydraLoRA/peft/shared.py"
    click ExpertB "https://github.com/clin0212/hydralora/blob/main/HydraLoRA/peft/tuners/mmoelora.py"
    click ExpertB "https://github.com/clin0212/hydralora/blob/main/HydraLoRA/peft/tuners/mmoeloraS.py"
    click Transformers "https://github.com/clin0212/hydralora/tree/main/HydraLoRA/transformers/"
    click MoERouter "https://github.com/clin0212/hydralora/blob/main/MLLM-HydraLoRA/moe.py"
    click MoEPatch "https://github.com/clin0212/hydralora/blob/main/MLLM-HydraLoRA/moe_patch/modeling_llama.py"
    click MultimodalBuilder "https://github.com/clin0212/hydralora/blob/main/MLLM-HydraLoRA/builder.py"
    click MultiWrapper "https://github.com/clin0212/hydralora/blob/main/MLLM-HydraLoRA/llava_llama.py"
    click MultiTrain "https://github.com/clin0212/hydralora/blob/main/MLLM-HydraLoRA/train.py"
    click ExportUtils "https://github.com/clin0212/hydralora/tree/main/MLLM-HydraLoRA/export/"
    click Motivation "https://github.com/clin0212/hydralora/blob/main/Motivation/tsen_lora.py"
    click Figures "https://github.com/clin0212/hydralora/tree/main/figures/"
    click EnvConfig "https://github.com/clin0212/hydralora/blob/main/environment.yml"
    click RootREADME "https://github.com/clin0212/hydralora/blob/main/README.md"

    %% Styles
    classDef data fill:#D0E6F5,stroke:#0366D6,color:#03396c
    classDef model fill:#D5F5E3,stroke:#239B56,color:#1E8449
    classDef routing fill:#FDEBD0,stroke:#DC7633,color:#B03A2E
    classDef external fill:#E5E7E9,stroke:#A6ACAF,color:#626567
```
