# sakanaai/self-adaptive-llms architecture

> Architecture diagram of the GitHub repository sakanaai/self-adaptive-llms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sakanaai/self-adaptive-llms
- Repository: https://github.com/sakanaai/self-adaptive-llms

## Overview

##### Project Type & Purpose
- Type: Research-driven machine learning framework for self-adaptive LLMs.
- Purpose: At training time, learn task-specific “expert” vectors via reinforcement learning; at inference time, dispatch on task properties then dynamically mix those vectors into a frozen base LLM.

##### High-Level Architecture Components
A. Entry Points (Shell scripts & Hydra)
- scripts/train_task_expert.sh → invokes `svd_reinforce_hydra.py` with Hydra configs for training.
- scripts/eval_{few_shot,prompt_based}.sh → invoke evaluation pipeline in `evaluation/fishfarm`.
B. Configuration System (Hydra)
- Directory: `cfgs/`
- Subfolders: base_model, mode (training vs. eval), optimization, policy, task.
- Role: Declaratively wire together models, tasks, policies, and optimizers.
C. Core Modules
1. Base Model
- Location: `base_model/`
- Contains wrappers for off-the-shelf LLMs (Llama3Instruct, Mistral03Instruct).
- Role: Encapsulate forward passes, tokenization, device placement.
2. Policy Modules
- Location: `policy/`
- Classes: `base.py`, `weighted_combination.py`
- Role: Define how expert vectors are combined and applied to the base model.
3. Optimization Modules
- Location: `optim_modules.py` & `cfgs/optimization/`
- Algorithms: CEM, Reinforce, RSM.
- Role: Given rewards, update the expert vector parameters.
4. Task Definitions
- Location: top-level `tasks/` & `cfgs/task/`
- Role: Provide prompt templates, datasets, and evaluation logic for each NLP task.
D. Training Pipeline (Two-Pass Mechanism)
1. Dispatch Pass:
- Inspect the incoming prompt/task metadata.
- Select the appropriate expert or a combination policy.
2. Adaptation Pass:
- Mix the expert vector(s) into the base model’s weights.
- Execute a forward pass, collect model outputs.
- Compute a task-specific reward.
- Hand reward to the chosen optimization algorithm to update the expert.
3. Persistence:
- Save trained expert vectors to disk.
E. Evaluation Pipeline
- Uses the `evaluation/fishfarm` package.
- Loads saved experts & base model via `fishfarm.models`.
- Executes few-shot or prompt-based evaluation tasks in `fishfarm.tasks`.
- Emits performance metrics and logs.

##### Relationships & Data Flows
- Scripts → Hydra Config → instantiate Modules (BaseModel, Policy, Optimizer, Task)
- Task module → generates prompts/data → sent to BaseModel + Policy adaptor
- BaseModel (+ adapted weights) → produces output → Task evaluates → reward
- Reward → Optimizer → updates expert vector → loop
- Upon convergence → Expert vectors saved → consumed by Evaluation pipeline
- Evaluation pipeline follows similar flow, minus optimizer, and records metrics.

##### Architectural Patterns & Principles
- Modular, plugin-style components wired via Hydra (separation of concerns).
- Two-pass inference (dispatch + adaptation) decouples task detection from model execution.
- Reinforcement learning loop abstracted via optim_modules.
- Config-driven experimentation for reproducibility.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Points
    subgraph "Entry Points"
        EP1("train_task_expert.sh"):::script
        EP2("eval_few_shot.sh"):::script
        EP3("eval_prompt_based.sh"):::script
        EP4("svd_reinforce_hydra.py"):::script
    end

    %% Config System
    subgraph "Config System (Hydra)"
        CFG_MAIN("config.yaml"):::config
        subgraph "base_model"
            BM1("llama3i8b.yaml"):::config
            BM2("mistral03i7b.yaml"):::config
        end
        subgraph "mode"
            MD1("training.yaml"):::config
            MD2("eval.yaml"):::config
        end
        subgraph "optimization"
            OPT1("cem.yaml"):::config
            OPT2("reinforce.yaml"):::config
            OPT3("rsm.yaml"):::config
        end
        subgraph "policy"
            P1("default.yaml"):::config
            P2("wcomb.yaml"):::config
        end
        subgraph "task"
            TASK_CFG("task YAMLs"):::config
        end
    end

    %% Core Modules
    subgraph "Core Modules"
        subgraph "Base Model"
            BM_INIT("__init__.py"):::base
            BM_BASE("base.py"):::base
            BM_LLA("llama3instruct.py"):::base
            BM_MIS("mistral03instruct.py"):::base
        end
        subgraph "Policy Modules"
            POL_INIT("__init__.py"):::policy
            POL_BASE("base.py"):::policy
            POL_W("weighted_combination.py"):::policy
        end
        subgraph "Optimization Modules"
            OPT_MOD("optim_modules.py"):::optim
        end
        subgraph "Task Definitions"
            TS_INIT("__init__.py"):::task
            TS_BASE("base.py"):::task
            TS_ARC("arc.py"):::task
            TS_CLS("cls.py"):::task
            TS_GSM("gsm8k.py"):::task
            TS_MATH("math.py"):::task
            TS_MBPP("mbpp2.py"):::task
            TS_CFGS("cfgs/task/"):::config
        end
    end

    %% Storage
    StorageVec(["Saved Expert Vectors"]):::storage
    StorageLogs(["Logs"]):::storage

    %% Training Flow
    EP1 --> EP4
    EP2 --> EP4
    EP3 --> EP4
    EP4 --> CFG_MAIN
    CFG_MAIN --> BM_BASE
    CFG_MAIN --> POL_BASE
    CFG_MAIN --> OPT_MOD
    CFG_MAIN --> TS_BASE

    TS_BASE -->|"prompts"| POL_BASE
    POL_BASE -->|"applies expert vectors"| BM_BASE
    BM_BASE -->|"outputs"| TS_BASE
    TS_BASE -->|"reward"| OPT_MOD
    OPT_MOD -->|"updates expert"| POL_BASE
    OPT_MOD --> StorageVec
    OPT_MOD --> StorageLogs

    %% Evaluation Pipeline
    subgraph "Evaluation Pipeline (fishfarm)"
        EVAL_PY("pyproject.toml"):::eval
        TOX("tox.ini"):::eval
        subgraph "fishfarm package"
            FF_INIT("__init__.py"):::eval
            FF_CHAT("chat_templates.py"):::eval
            FF_IMP("imports.py"):::eval
            FF_LOG("logging.py"):::eval
            subgraph "models"
                FM_BASE("base.py"):::eval
                FM_TOKEN("tokenization_utils.py"):::eval
                FM_VLLM("vllm_model.py"):::eval
            end
            subgraph "tasks"
                FT_BASE("base.py"):::eval
                FT_AI2("ai2_arc.py"):::eval
                FT_COMP("competation_math.py"):::eval
                FT_LANG("language_restricted_math.py"):::eval
                subgraph "evalplus"
                    EPALL("sub-modules"):::eval
                end
            end
        end
    end

    StorageVec --> EVAL_PY
    StorageVec --> FF_INIT
    FF_INIT --> FM_BASE
    FM_BASE --> FT_BASE
    FT_BASE --> FT_AI2
    FT_BASE --> FT_COMP
    FT_BASE --> FT_LANG
    FT_LANG --> EPALL

    %% Click events
    click EP1 "https://github.com/sakanaai/self-adaptive-llms/blob/main/scripts/train_task_expert.sh"
    click EP2 "https://github.com/sakanaai/self-adaptive-llms/blob/main/scripts/eval_few_shot.sh"
    click EP3 "https://github.com/sakanaai/self-adaptive-llms/blob/main/scripts/eval_prompt_based.sh"
    click EP4 "https://github.com/sakanaai/self-adaptive-llms/blob/main/svd_reinforce_hydra.py"
    click CFG_MAIN "https://github.com/sakanaai/self-adaptive-llms/blob/main/cfgs/config.yaml"
    click BM1 "https://github.com/sakanaai/self-adaptive-llms/blob/main/cfgs/base_model/llama3i8b.yaml"
    click BM2 "https://github.com/sakanaai/self-adaptive-llms/blob/main/cfgs/base_model/mistral03i7b.yaml"
    click MD1 "https://github.com/sakanaai/self-adaptive-llms/blob/main/cfgs/mode/training.yaml"
    click MD2 "https://github.com/sakanaai/self-adaptive-llms/blob/main/cfgs/mode/eval.yaml"
    click OPT1 "https://github.com/sakanaai/self-adaptive-llms/blob/main/cfgs/optimization/cem.yaml"
    click OPT2 "https://github.com/sakanaai/self-adaptive-llms/blob/main/cfgs/optimization/reinforce.yaml"
    click OPT3 "https://github.com/sakanaai/self-adaptive-llms/blob/main/cfgs/optimization/rsm.yaml"
    click P1 "https://github.com/sakanaai/self-adaptive-llms/blob/main/cfgs/policy/default.yaml"
    click P2 "https://github.com/sakanaai/self-adaptive-llms/blob/main/cfgs/policy/wcomb.yaml"
    click TASK_CFG "https://github.com/sakanaai/self-adaptive-llms/tree/main/cfgs/task/"
    click BM_INIT "https://github.com/sakanaai/self-adaptive-llms/blob/main/base_model/__init__.py"
    click BM_BASE "https://github.com/sakanaai/self-adaptive-llms/blob/main/base_model/base.py"
    click BM_LLA "https://github.com/sakanaai/self-adaptive-llms/blob/main/base_model/llama3instruct.py"
    click BM_MIS "https://github.com/sakanaai/self-adaptive-llms/blob/main/base_model/mistral03instruct.py"
    click POL_INIT "https://github.com/sakanaai/self-adaptive-llms/blob/main/policy/__init__.py"
    click POL_BASE "https://github.com/sakanaai/self-adaptive-llms/blob/main/policy/base.py"
    click POL_W "https://github.com/sakanaai/self-adaptive-llms/blob/main/policy/weighted_combination.py"
    click OPT_MOD "https://github.com/sakanaai/self-adaptive-llms/blob/main/optim_modules.py"
    click TS_INIT "https://github.com/sakanaai/self-adaptive-llms/blob/main/tasks/__init__.py"
    click TS_BASE "https://github.com/sakanaai/self-adaptive-llms/blob/main/tasks/base.py"
    click TS_ARC "https://github.com/sakanaai/self-adaptive-llms/blob/main/tasks/arc.py"
    click TS_CLS "https://github.com/sakanaai/self-adaptive-llms/blob/main/tasks/cls.py"
    click TS_GSM "https://github.com/sakanaai/self-adaptive-llms/blob/main/tasks/gsm8k.py"
    click TS_MATH "https://github.com/sakanaai/self-adaptive-llms/blob/main/tasks/math.py"
    click TS_MBPP "https://github.com/sakanaai/self-adaptive-llms/blob/main/tasks/mbpp2.py"
    click EVAL_PY "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/pyproject.toml"
    click TOX "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/tox.ini"
    click FF_INIT "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/fishfarm/__init__.py"
    click FF_CHAT "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/fishfarm/chat_templates.py"
    click FF_IMP "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/fishfarm/imports.py"
    click FF_LOG "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/fishfarm/logging.py"
    click FM_BASE "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/fishfarm/models/base.py"
    click FM_TOKEN "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/fishfarm/models/tokenization_utils.py"
    click FM_VLLM "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/fishfarm/models/vllm_model.py"
    click FT_BASE "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/fishfarm/tasks/base.py"
    click FT_AI2 "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/fishfarm/tasks/ai2_arc.py"
    click FT_COMP "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/fishfarm/tasks/competation_math.py"
    click FT_LANG "https://github.com/sakanaai/self-adaptive-llms/blob/main/evaluation/fishfarm/fishfarm/tasks/language_restricted_math.py"
    click EPALL "https://github.com/sakanaai/self-adaptive-llms/tree/main/evaluation/fishfarm/fishfarm/tasks/evalplus/"

    %% Styles
    classDef script fill:#f4f4f4,stroke:#333,stroke-width:1px
    classDef config fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
    classDef base fill:#bbdefb,stroke:#1565c0,stroke-width:1px
    classDef policy fill:#fff9c4,stroke:#f9a825,stroke-width:1px
    classDef optim fill:#ffe0b2,stroke:#ef6c00,stroke-width:1px
    classDef task fill:#e1bee7,stroke:#6a1b9a,stroke-width:1px
    classDef storage fill:#eceff1,stroke:#607d8b,stroke-width:1px,stroke-dasharray: 5 5
    classDef eval fill:#ffcdd2,stroke:#b71c1c,stroke-width:1px
```
