# xiaozi-max/lr-matters-lora architecture

> Architecture diagram of the GitHub repository xiaozi-max/lr-matters-lora, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xiaozi-max/lr-matters-lora
- Repository: https://github.com/xiaozi-max/lr-matters-lora

## Overview

This repository is a research codebase for comparing LoRA fine-tuning methods and learning-rate choices. The main workflow is to train adapters on language-model tasks, evaluate model outputs, and aggregate results across methods and learning rates. Separate analysis estimates Hessian curvature, while the paper list and practical heuristics preserve the study’s literature-review and tuning-guidance branches. README documents the major subsystems; sampled source supports training, adapter generation, result aggregation, and Hessian analysis, but does not establish every evaluation or data-preparation connection.

Main entry points: [`run-lora/train.py`](https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-lora/train.py) (Fine-tune models), [`run-hessian/lanczos.py`](https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-hessian/lanczos.py) (Hessian estimation), [`run-lora/utils/gen_vllm_lora`](https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-lora/utils/gen_vllm_lora) (Task generation), [`run-lora/utils`](https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-lora/utils) (Evaluation tools)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["LoRA experiments"]
  node_train["Fine-tune models<br/>[train.py]"]
  node_methods["LoRA variants"]
  node_optimizers["LoFT optimizer<br/>[optimizer.py]"]
  node_adapters[("Adapter checkpoints")]
  node_generation["Task generation"]
  node_eval["Evaluation tools"]
  node_records[("Experiment records<br/>[results.csv]")]
  node_summary["Result summaries<br/>[summary.py]"]
  node_result_artifacts[("Evaluation outputs")]
end

subgraph group_analysis["Curvature analysis"]
  node_hessian["Hessian estimation<br/>[lanczos.py]"]
  node_hessian_data["Hessian utilities"]
  node_eigenresults[("Eigenvalue results")]
end

subgraph group_research["Study evidence"]
  node_papers["LoRA paper list<br/>[README.md]"]
end

subgraph group_guidance["Tuning guidance"]
  node_heuristics["Tuning heuristics<br/>[README.md]"]
end

node_researcher(("Researcher"))
node_datasets[("Task datasets")]
node_models[("Base models")]

node_researcher -->|"runs training"| node_train
node_datasets -->|"provides data"| node_train
node_models -->|"provides model"| node_train
node_train -->|"configures variants"| node_methods
node_train -->|"uses optimizer"| node_optimizers
node_train -->|"saves adapters"| node_adapters
node_researcher -->|"runs generation"| node_generation
node_models -->|"provides model"| node_generation
node_adapters -->|"loads adapter"| node_generation
node_datasets -->|"provides prompts"| node_generation
node_generation -->|"writes responses"| node_result_artifacts
node_researcher -->|"runs evaluation"| node_eval
node_result_artifacts -->|"provides responses"| node_eval
node_eval -->|"records metrics"| node_records
node_records -->|"reads results"| node_summary
node_researcher -->|"runs analysis"| node_hessian
node_models -->|"provides model"| node_hessian
node_datasets -->|"provides data"| node_hessian
node_hessian -->|"loads data helpers"| node_hessian_data
node_hessian -->|"writes eigenvalues"| node_eigenresults
node_researcher -->|"reviews literature"| node_papers
node_researcher -->|"consults guidance"| node_heuristics

click node_train "https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-lora/train.py"
click node_methods "https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-lora/utils"
click node_optimizers "https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-lora/utils/LoFT/loft_optim/optimizer.py"
click node_generation "https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-lora/utils/gen_vllm_lora"
click node_eval "https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-lora/utils"
click node_records "https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-lora/exp_record/results.csv"
click node_summary "https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-lora/exp_record/summary.py"
click node_hessian "https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-hessian/lanczos.py"
click node_hessian_data "https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-hessian/utils"
click node_eigenresults "https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-hessian/eigen_results"
click node_papers "https://github.com/xiaozi-max/lr-matters-lora/blob/dev/awesome-lora-paper-list/README.md"
click node_heuristics "https://github.com/xiaozi-max/lr-matters-lora/blob/dev/practical-heuristics/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_train,node_methods,node_optimizers,node_adapters,node_generation,node_eval,node_records,node_summary,node_result_artifacts toneBlue
class node_hessian,node_hessian_data,node_eigenresults,node_datasets,node_models toneAmber
class node_papers toneMint
class node_heuristics toneRose
class node_researcher toneIndigo
```

## Components

### LoRA experiments

- [Fine-tune models](https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-lora/train.py): `run-lora/train.py`
- [LoRA variants](https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-lora/utils): `run-lora/utils`
- [LoFT optimizer](https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-lora/utils/LoFT/loft_optim/optimizer.py): `run-lora/utils/LoFT/loft_optim/optimizer.py`
- **Adapter checkpoints**
- [Task generation](https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-lora/utils/gen_vllm_lora): `run-lora/utils/gen_vllm_lora`
- [Evaluation tools](https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-lora/utils): `run-lora/utils`
- [Experiment records](https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-lora/exp_record/results.csv): `run-lora/exp_record/results.csv`
- [Result summaries](https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-lora/exp_record/summary.py): `run-lora/exp_record/summary.py`
- **Evaluation outputs**

### Curvature analysis

- [Hessian estimation](https://github.com/xiaozi-max/lr-matters-lora/blob/dev/run-hessian/lanczos.py): `run-hessian/lanczos.py`
- [Hessian utilities](https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-hessian/utils): `run-hessian/utils`
- [Eigenvalue results](https://github.com/xiaozi-max/lr-matters-lora/tree/dev/run-hessian/eigen_results): `run-hessian/eigen_results`

### Study evidence

- [LoRA paper list](https://github.com/xiaozi-max/lr-matters-lora/blob/dev/awesome-lora-paper-list/README.md): `awesome-lora-paper-list/README.md`

### Tuning guidance

- [Tuning heuristics](https://github.com/xiaozi-max/lr-matters-lora/blob/dev/practical-heuristics/README.md): `practical-heuristics/README.md`

### Other components

- **Researcher**
- **Task datasets**
- **Base models**

## Connections

- Researcher → Fine-tune models: runs training
- Task datasets → Fine-tune models: provides data
- Base models → Fine-tune models: provides model
- Fine-tune models → LoRA variants: configures variants
- Fine-tune models → LoFT optimizer: uses optimizer
- Fine-tune models → Adapter checkpoints: saves adapters
- Researcher → Task generation: runs generation
- Base models → Task generation: provides model
- Adapter checkpoints → Task generation: loads adapter
- Task datasets → Task generation: provides prompts
- Task generation → Evaluation outputs: writes responses
- Researcher → Evaluation tools: runs evaluation
- Evaluation outputs → Evaluation tools: provides responses
- Evaluation tools → Experiment records: records metrics
- Experiment records → Result summaries: reads results
- Researcher → Hessian estimation: runs analysis
- Base models → Hessian estimation: provides model
- Task datasets → Hessian estimation: provides data
- Hessian estimation → Hessian utilities: loads data helpers
- Hessian estimation → Eigenvalue results: writes eigenvalues
- Researcher → LoRA paper list: reviews literature
- Researcher → Tuning heuristics: consults guidance