How devarenidhi25/toy-rlhf works
toyrlhf is a small research implementation for comparing tabular and neural policies on a toy RLHF objective. The workflow generates teacher samples, fits a reference policy, then optimizes it against a reward while tracking exact-enumeration metrics. Separate experiments cover REINFORCE baselines, actor-critic/PPO, and neural PPO. The excerpts show core algorithms and module dependencies; experiment-script links are supported by the source index, but README documentation is absent.
Main entry points: scripts Experiment scripts · sequences.py Sequence utilities · policy.py Tabular policy · corpus.py Teacher corpus
Components 12
Data and objective
- Toy configuration
src/toyrlhf/config.py - Sequence utilities
src/toyrlhf/sequences.py - Teacher corpus
src/toyrlhf/corpus.py - Reward function
src/toyrlhf/reward.py - Exact evaluator
src/toyrlhf/exact.py
Tabular policies
- Tabular policy
src/toyrlhf/policy.py - Tabular pretraining
src/toyrlhf/pretrain.py
Policy optimization
- REINFORCE
src/toyrlhf/reinforce.py - Actor-critic and PPO
src/toyrlhf/actor_critic.py
Neural policies
- Neural policy
src/toyrlhf/neural.py
Other components
- Researcher
- Experiment scripts
scripts
Connections 21
- Researcher → Experiment scripts: runs experiments
- Experiment scripts → Toy configuration: loads settings
scripts/01_pretrain_reference.py - Experiment scripts → Teacher corpus: creates samples
scripts/01_pretrain_reference.py - Experiment scripts → Tabular pretraining: fits reference
scripts/01_pretrain_reference.py - Experiment scripts → Exact evaluator: evaluates policies
scripts/01_pretrain_reference.py - Experiment scripts → Tabular policy: sets up policy
scripts/02_reinforce.py - Experiment scripts → Reward function: supplies reward
scripts/02_reinforce.py - Experiment scripts → REINFORCE: runs REINFORCE
scripts/02_reinforce.py - Experiment scripts → Actor-critic and PPO: runs PPO
scripts/03_actor_critic_ppo.py - Experiment scripts → Neural policy: runs neural PPO
scripts/04_neural_ppo.py - Teacher corpus → Tabular policy: samples from
src/toyrlhf/corpus.py - Teacher corpus → Sequence utilities: uses prefix indexing
src/toyrlhf/corpus.py - Tabular policy → Sequence utilities: uses prefix indexing
src/toyrlhf/policy.py - Tabular pretraining → Tabular policy: fits logits
src/toyrlhf/pretrain.py - Tabular pretraining → Sequence utilities: counts prefixes
src/toyrlhf/pretrain.py - Reward function → Toy configuration: reads reward settings
src/toyrlhf/reward.py - Exact evaluator → Sequence utilities: enumerates sequences
src/toyrlhf/exact.py - Exact evaluator → Reward function: computes rewards
src/toyrlhf/exact.py - REINFORCE → Sequence utilities: indexes prefixes
src/toyrlhf/reinforce.py - Actor-critic and PPO → Sequence utilities: indexes prefixes
src/toyrlhf/actor_critic.py - Neural policy → Actor-critic and PPO: uses GAE
src/toyrlhf/neural.py