How xiaozi-max/lr-matters-lora works
This repository is a research codebase for comparing LoRA fine-tuning methods and learning-rate choices. The main workflow is to train adapters on language-model tasks, evaluate model outputs, and aggregate results across methods and learning rates. Separate analysis estimates Hessian curvature, while the paper list and practical heuristics preserve the study’s literature-review and tuning-guidance branches. README documents the major subsystems; sampled source supports training, adapter generation, result aggregation, and Hessian analysis, but does not establish every evaluation or data-preparation connection.
Main entry points: train.py Fine-tune models · lanczos.py Hessian estimation · gen_vllm_lora Task generation · utils Evaluation tools
Components 17
LoRA experiments
- Fine-tune models
run-lora/train.py - LoRA variants
run-lora/utils - LoFT optimizer
run-lora/utils/LoFT/loft_optim/optimizer.py - Adapter checkpoints
- Task generation
run-lora/utils/gen_vllm_lora - Evaluation tools
run-lora/utils - Experiment records
run-lora/exp_record/results.csv - Result summaries
run-lora/exp_record/summary.py - Evaluation outputs
Curvature analysis
- Hessian estimation
run-hessian/lanczos.py - Hessian utilities
run-hessian/utils - Eigenvalue results
run-hessian/eigen_results
Study evidence
- LoRA paper list
awesome-lora-paper-list/README.md
Tuning guidance
- Tuning heuristics
practical-heuristics/README.md
Other components
- Researcher
- Task datasets
- Base models
Connections 22
- Researcher → Fine-tune models: runs training
- Task datasets → Fine-tune models: provides data
- Base models → Fine-tune models: provides model
- Fine-tune models → LoRA variants: configures variants
- Fine-tune models → LoFT optimizer: uses optimizer
- Fine-tune models → Adapter checkpoints: saves adapters
- Researcher → Task generation: runs generation
- Base models → Task generation: provides model
- Adapter checkpoints → Task generation: loads adapter
- Task datasets → Task generation: provides prompts
- Task generation → Evaluation outputs: writes responses
- Researcher → Evaluation tools: runs evaluation
- Evaluation outputs → Evaluation tools: provides responses
- Evaluation tools → Experiment records: records metrics
- Experiment records → Result summaries: reads results
- Researcher → Hessian estimation: runs analysis
- Base models → Hessian estimation: provides model
- Task datasets → Hessian estimation: provides data
- Hessian estimation → Hessian utilities: loads data helpers
- Hessian estimation → Eigenvalue results: writes eigenvalues
- Researcher → LoRA paper list: reviews literature
- Researcher → Tuning heuristics: consults guidance