# yasaisen/hardprompt2softprompt architecture

> Architecture diagram of the GitHub repository yasaisen/hardprompt2softprompt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yasaisen/hardprompt2softprompt
- Repository: https://github.com/yasaisen/hardprompt2softprompt

## Overview

##### Project type and purpose
- This is an open-source reinforcement-learning fine-tuning framework for large language models (LLMs).
- It performs Proximal Policy Optimization (PPO) on a prefix-tuned Italian LM (“gemma-it”), combining a policy model (prefix tuning) with learned reward and value heads.

##### File structure
Top-level files and directories:
- LICENSE, README.md, __init__.py
- common: utility functions (common/utils.py) shared across modules
- datasets: data ingestion and preprocessing for RL (RMv2, singleStepPPO)
- models:
– policyModel: prefix-tuning policy network wrappers
– rewardModel: reward predictor network
– valueHead: value function estimator
- projects: experiment YAML configurations for training runs
- trainer:
– singleStepPPOTrainer: orchestrates data-model-RL loop
– building_* scripts: construction of trainer instances

##### README for additional insights
- Purpose: “do PPO on gemma-it based lm with prefix tuning”
- Dependencies: PyTorch, Transformers
- Usage: import PrefixTuningPolicyModel, generate responses with prefix, load pre-trained weights

a. Main components
1. Configuration Manager: reads YAML files from projects/
2. Data Pipeline: datasets/*.py for preparing (context, prompts, rewards)
3. Policy Module: PrefixTuningPolicyModel (models/policyModel)
4. Reward Module: RewardModel (models/rewardModel)
5. Value Module: ValueHead (models/valueHead)
6. Trainer Orchestrator: singleStepPPOTrainer
7. Utilities: common/utils (tokenization, logging, checkpointing)
8. External Frameworks: PyTorch, Transformers
9. Storage & Checkpoints: local outputs folder, optional GCP/Drive links
b. Relationships and interactions
- The Configuration Manager passes hyperparameters to the Trainer.
- The Trainer requests data batches from the Data Pipeline.
- For each batch, the Trainer invokes the Policy Module to sample actions (prefix tokens).
- The sampled output is scored by the Reward Module.
- The Value Module estimates baseline values.
- Trainer computes PPO loss (policy, value, entropy), updates policy/value heads.
- Utilities handle tokenization, logging metrics, and checkpoint saving.
c. Architectural patterns and design principles
- Modular separation: policy, reward, and value networks in isolated packages.
- YAML-driven configuration for reproducible experiments.
- Trainer as a high-level orchestrator implementing the RL loop.
- Reusable utilities for common tasks (e.g., tokenization, I/O).
d. Key technologies and libraries
- PyTorch 2.x for model definition and training loops.
- Hugging Face Transformers for LM backbones and tokenizers.
- Prefix tuning implementation wrapping base LM.

##### Detail
- Module’s inputs and outputs.
- How common/utils functions (tokenize, logging) are invoked.
- Experiment config ingestion, checkpoint read/write, and optional external storage.
- Don’t over-abstract—break the system into as many logical components as needed for clarity.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configuration Layer"
        Config["Configuration Manager"]:::config
    end

    subgraph "Data Layer"
        RMv2["RMv2 Dataset"]:::data
        PPOData["single-Step PPO Dataset"]:::data
    end

    subgraph "Model Components"
        Policy["Policy Module"]:::policy
        Reward["Reward Module"]:::reward
        Value["Value Module"]:::value
    end

    subgraph "Trainer Orchestrator"
        Trainer["Single-Step PPO Trainer"]:::trainer
        RewardTrainer["Reward-Only Trainer Builder"]:::trainer
    end

    Utils["Utilities"]:::utils

    subgraph "External Frameworks"
        PyTorch["PyTorch"]:::external
        Transformers["Transformers"]:::external
    end

    Storage["Storage & Checkpoints"]:::storage

    %% Relationships
    Config -->|"loads YAML configs"| Trainer
    Config -->|"loads YAML configs"| RewardTrainer
    Trainer -->|"requests batches"| RMv2
    Trainer -->|"requests batches"| PPOData
    RewardTrainer -->|"requests batches"| RMv2
    RewardTrainer -->|"requests batches"| PPOData
    Trainer -->|"samples actions"| Policy
    Trainer -->|"scores outputs"| Reward
    Trainer -->|"estimates values"| Value
    Trainer -->|"uses utils"| Utils
    Utils -->|"saves checkpoints"| Storage
    Trainer -->|"writes checkpoints"| Storage
    Policy -->|"model defs"| Transformers
    Reward -->|"training ops"| PyTorch
    Value -->|"training ops"| PyTorch

    %% Click Events
    click Config "https://github.com/yasaisen/hardprompt2softprompt/tree/main/projects/"
    click RMv2 "https://github.com/yasaisen/hardprompt2softprompt/blob/main/datasets/RMv2_dataset.py"
    click PPOData "https://github.com/yasaisen/hardprompt2softprompt/blob/main/datasets/singleStepPPO_v1.py"
    click Policy "https://github.com/yasaisen/hardprompt2softprompt/blob/main/models/policyModel/modeling_policyModel.py"
    click Reward "https://github.com/yasaisen/hardprompt2softprompt/blob/main/models/rewardModel/modeling_rewardModel.py"
    click Value "https://github.com/yasaisen/hardprompt2softprompt/blob/main/models/valueHead/modeling_valueHead.py"
    click Trainer "https://github.com/yasaisen/hardprompt2softprompt/blob/main/trainer/singleStepPPOTrainer/building_singleStepPPOTrainer.py"
    click RewardTrainer "https://github.com/yasaisen/hardprompt2softprompt/blob/main/trainer/singleStepPPOTrainer/building_RewardModelTrainer.py"
    click Utils "https://github.com/yasaisen/hardprompt2softprompt/blob/main/common/utils.py"

    %% Styles
    classDef config fill:#f2f2f2,stroke:#333,stroke-width:1px
    classDef data fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef policy fill:#d4edda,stroke:#333,stroke-width:1px
    classDef reward fill:#fff3cd,stroke:#333,stroke-width:1px
    classDef value fill:#e2d6f3,stroke:#333,stroke-width:1px
    classDef trainer fill:#d6d8db,stroke:#333,stroke-width:1px
    classDef utils fill:#f0f0f0,stroke:#333,stroke-width:1px
    classDef external fill:#fff5cc,stroke:#333,stroke-width:1px
    classDef storage fill:#d1ecf1,stroke:#333,stroke-width:1px
```
