How sakanaai/self-adaptive-llms works
Project Type & Purpose
• Type: Research-driven machine learning framework for self-adaptive LLMs.
• Purpose: At training time, learn task-specific “expert” vectors via reinforcement learning; at inference time, dispatch on task properties then dynamically mix those vectors into a frozen base LLM.
High-Level Architecture Components
A. Entry Points (Shell scripts & Hydra)
• scripts/train_task_expert.sh → invokes svd_reinforce_hydra.py with Hydra configs for training.
• scripts/eval_{few_shot,prompt_based}.sh → invoke evaluation pipeline in evaluation/fishfarm.
B. Configuration System (Hydra)
• Directory: cfgs/
• Subfolders: base_model, mode (training vs. eval), optimization, policy, task.
• Role: Declaratively wire together models, tasks, policies, and optimizers.
C. Core Modules
1. Base Model
• Location: base_model/
• Contains wrappers for off-the-shelf LLMs (Llama3Instruct, Mistral03Instruct).
• Role: Encapsulate forward passes, tokenization, device placement.
2. Policy Modules
• Location: policy/
• Classes: base.py, weighted_combination.py
• Role: Define how expert vectors are combined and applied to the base model.
3. Optimization Modules
• Location: optim_modules.py & cfgs/optimization/
• Algorithms: CEM, Reinforce, RSM.
• Role: Given rewards, update the expert vector parameters.
4. Task Definitions
• Location: top-level tasks/ & cfgs/task/
• Role: Provide prompt templates, datasets, and evaluation logic for each NLP task.
D. Training Pipeline (Two-Pass Mechanism)
1. Dispatch Pass:
• Inspect the incoming prompt/task metadata.
• Select the appropriate expert or a combination policy.
2. Adaptation Pass:
• Mix the expert vector(s) into the base model’s weights.
• Execute a forward pass, collect model outputs.
• Compute a task-specific reward.
• Hand reward to the chosen optimization algorithm to update the expert.
3. Persistence:
• Save trained expert vectors to disk.
E. Evaluation Pipeline
• Uses the evaluation/fishfarm package.
• Loads saved experts & base model via fishfarm.models.
• Executes few-shot or prompt-based evaluation tasks in fishfarm.tasks.
• Emits performance metrics and logs.
Relationships & Data Flows
• Scripts → Hydra Config → instantiate Modules (BaseModel, Policy, Optimizer, Task)
• Task module → generates prompts/data → sent to BaseModel + Policy adaptor
• BaseModel (+ adapted weights) → produces output → Task evaluates → reward
• Reward → Optimizer → updates expert vector → loop
• Upon convergence → Expert vectors saved → consumed by Evaluation pipeline
• Evaluation pipeline follows similar flow, minus optimizer, and records metrics.
Architectural Patterns & Principles
• Modular, plugin-style components wired via Hydra (separation of concerns).
• Two-pass inference (dispatch + adaptation) decouples task detection from model execution.
• Reinforcement learning loop abstracted via optim_modules.
• Config-driven experimentation for reproducibility.