How yasaisen/hardprompt2softprompt works
Project type and purpose
• This is an open-source reinforcement-learning fine-tuning framework for large language models (LLMs).
• It performs Proximal Policy Optimization (PPO) on a prefix-tuned Italian LM (“gemma-it”), combining a policy model (prefix tuning) with learned reward and value heads.
File structure
Top-level files and directories:
• LICENSE, README.md, __init__.py
• common: utility functions (common/utils.py) shared across modules
• datasets: data ingestion and preprocessing for RL (RMv2, singleStepPPO)
• models:
– policyModel: prefix-tuning policy network wrappers
– rewardModel: reward predictor network
– valueHead: value function estimator
• projects: experiment YAML configurations for training runs
• trainer:
– singleStepPPOTrainer: orchestrates data-model-RL loop
– building_* scripts: construction of trainer instances
README for additional insights
• Purpose: “do PPO on gemma-it based lm with prefix tuning”
• Dependencies: PyTorch, Transformers
• Usage: import PrefixTuningPolicyModel, generate responses with prefix, load pre-trained weights
a. Main components
1. Configuration Manager: reads YAML files from projects/
2. Data Pipeline: datasets/*.py for preparing (context, prompts, rewards)
3. Policy Module: PrefixTuningPolicyModel (models/policyModel)
4. Reward Module: RewardModel (models/rewardModel)
5. Value Module: ValueHead (models/valueHead)
6. Trainer Orchestrator: singleStepPPOTrainer
7. Utilities: common/utils (tokenization, logging, checkpointing)
8. External Frameworks: PyTorch, Transformers
9. Storage & Checkpoints: local outputs folder, optional GCP/Drive links
b. Relationships and interactions
• The Configuration Manager passes hyperparameters to the Trainer.
• The Trainer requests data batches from the Data Pipeline.
• For each batch, the Trainer invokes the Policy Module to sample actions (prefix tokens).
• The sampled output is scored by the Reward Module.
• The Value Module estimates baseline values.
• Trainer computes PPO loss (policy, value, entropy), updates policy/value heads.
• Utilities handle tokenization, logging metrics, and checkpoint saving.
c. Architectural patterns and design principles
• Modular separation: policy, reward, and value networks in isolated packages.
• YAML-driven configuration for reproducible experiments.
• Trainer as a high-level orchestrator implementing the RL loop.
• Reusable utilities for common tasks (e.g., tokenization, I/O).
d. Key technologies and libraries
• PyTorch 2.x for model definition and training loops.
• Hugging Face Transformers for LM backbones and tokenizers.
• Prefix tuning implementation wrapping base LM.
Detail
• Module’s inputs and outputs.
• How common/utils functions (tokenize, logging) are invoked.
• Experiment config ingestion, checkpoint read/write, and optional external storage.
• Don’t over-abstract—break the system into as many logical components as needed for clarity.