# devarenidhi25/toy-rlhf architecture

> Architecture diagram of the GitHub repository devarenidhi25/toy-rlhf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/devarenidhi25/toy-rlhf
- Repository: https://github.com/devarenidhi25/toy-rlhf

## Overview

toyrlhf is a small research implementation for comparing tabular and neural policies on a toy RLHF objective. The workflow generates teacher samples, fits a reference policy, then optimizes it against a reward while tracking exact-enumeration metrics. Separate experiments cover REINFORCE baselines, actor-critic/PPO, and neural PPO. The excerpts show core algorithms and module dependencies; experiment-script links are supported by the source index, but README documentation is absent.

Main entry points: [`scripts`](https://github.com/devarenidhi25/toy-rlhf/tree/main/scripts) (Experiment scripts), [`src/toyrlhf/sequences.py`](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/sequences.py) (Sequence utilities), [`src/toyrlhf/policy.py`](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/policy.py) (Tabular policy), [`src/toyrlhf/corpus.py`](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/corpus.py) (Teacher corpus)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data and objective"]
  node_config["Toy configuration<br/>[config.py]"]
  node_sequences["Sequence utilities<br/>[sequences.py]"]
  node_corpus["Teacher corpus<br/>[corpus.py]"]
  node_reward["Reward function<br/>[reward.py]"]
  node_exact["Exact evaluator<br/>[exact.py]"]
end

subgraph group_tabular["Tabular policies"]
  node_policy["Tabular policy<br/>[policy.py]"]
  node_pretrain["Tabular pretraining<br/>[pretrain.py]"]
end

subgraph group_learning["Policy optimization"]
  node_reinforce["REINFORCE<br/>[reinforce.py]"]
  node_actorcritic["Actor-critic and PPO<br/>[actor_critic.py]"]
end

subgraph group_neural["Neural policies"]
  node_neuralpolicy["Neural policy<br/>[neural.py]"]
end

node_researcher(("Researcher"))
node_experiments["Experiment scripts"]

node_researcher -.->|"runs experiments"| node_experiments
node_experiments -->|"loads settings"| node_config
node_experiments -->|"creates samples"| node_corpus
node_experiments -->|"fits reference"| node_pretrain
node_experiments -->|"evaluates policies"| node_exact
node_experiments -->|"sets up policy"| node_policy
node_experiments -->|"supplies reward"| node_reward
node_experiments -->|"runs REINFORCE"| node_reinforce
node_experiments -->|"runs PPO"| node_actorcritic
node_experiments -->|"runs neural PPO"| node_neuralpolicy
node_corpus -->|"samples from"| node_policy
node_corpus -->|"uses prefix indexing"| node_sequences
node_policy -->|"uses prefix indexing"| node_sequences
node_pretrain -->|"fits logits"| node_policy
node_pretrain -->|"counts prefixes"| node_sequences
node_reward -->|"reads reward settings"| node_config
node_exact -->|"enumerates sequences"| node_sequences
node_exact -->|"computes rewards"| node_reward
node_reinforce -->|"indexes prefixes"| node_sequences
node_actorcritic -->|"indexes prefixes"| node_sequences
node_neuralpolicy -->|"uses GAE"| node_actorcritic

click node_experiments "https://github.com/devarenidhi25/toy-rlhf/tree/main/scripts"
click node_config "https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/config.py"
click node_sequences "https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/sequences.py"
click node_corpus "https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/corpus.py"
click node_reward "https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/reward.py"
click node_exact "https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/exact.py"
click node_policy "https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/policy.py"
click node_pretrain "https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/pretrain.py"
click node_reinforce "https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/reinforce.py"
click node_actorcritic "https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/actor_critic.py"
click node_neuralpolicy "https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/neural.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_sequences,node_corpus,node_reward,node_exact toneBlue
class node_policy,node_pretrain toneAmber
class node_reinforce,node_actorcritic toneMint
class node_neuralpolicy toneRose
class node_researcher toneIndigo
class node_experiments toneTeal
```

## Components

### Data and objective

- [Toy configuration](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/config.py): `src/toyrlhf/config.py`
- [Sequence utilities](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/sequences.py): `src/toyrlhf/sequences.py`
- [Teacher corpus](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/corpus.py): `src/toyrlhf/corpus.py`
- [Reward function](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/reward.py): `src/toyrlhf/reward.py`
- [Exact evaluator](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/exact.py): `src/toyrlhf/exact.py`

### Tabular policies

- [Tabular policy](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/policy.py): `src/toyrlhf/policy.py`
- [Tabular pretraining](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/pretrain.py): `src/toyrlhf/pretrain.py`

### Policy optimization

- [REINFORCE](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/reinforce.py): `src/toyrlhf/reinforce.py`
- [Actor-critic and PPO](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/actor_critic.py): `src/toyrlhf/actor_critic.py`

### Neural policies

- [Neural policy](https://github.com/devarenidhi25/toy-rlhf/blob/main/src/toyrlhf/neural.py): `src/toyrlhf/neural.py`

### Other components

- **Researcher**
- [Experiment scripts](https://github.com/devarenidhi25/toy-rlhf/tree/main/scripts): `scripts`

## Connections

- Researcher → Experiment scripts: runs experiments
- Experiment scripts → Toy configuration: loads settings (evidence: `scripts/01_pretrain_reference.py`)
- Experiment scripts → Teacher corpus: creates samples (evidence: `scripts/01_pretrain_reference.py`)
- Experiment scripts → Tabular pretraining: fits reference (evidence: `scripts/01_pretrain_reference.py`)
- Experiment scripts → Exact evaluator: evaluates policies (evidence: `scripts/01_pretrain_reference.py`)
- Experiment scripts → Tabular policy: sets up policy (evidence: `scripts/02_reinforce.py`)
- Experiment scripts → Reward function: supplies reward (evidence: `scripts/02_reinforce.py`)
- Experiment scripts → REINFORCE: runs REINFORCE (evidence: `scripts/02_reinforce.py`)
- Experiment scripts → Actor-critic and PPO: runs PPO (evidence: `scripts/03_actor_critic_ppo.py`)
- Experiment scripts → Neural policy: runs neural PPO (evidence: `scripts/04_neural_ppo.py`)
- Teacher corpus → Tabular policy: samples from (evidence: `src/toyrlhf/corpus.py`)
- Teacher corpus → Sequence utilities: uses prefix indexing (evidence: `src/toyrlhf/corpus.py`)
- Tabular policy → Sequence utilities: uses prefix indexing (evidence: `src/toyrlhf/policy.py`)
- Tabular pretraining → Tabular policy: fits logits (evidence: `src/toyrlhf/pretrain.py`)
- Tabular pretraining → Sequence utilities: counts prefixes (evidence: `src/toyrlhf/pretrain.py`)
- Reward function → Toy configuration: reads reward settings (evidence: `src/toyrlhf/reward.py`)
- Exact evaluator → Sequence utilities: enumerates sequences (evidence: `src/toyrlhf/exact.py`)
- Exact evaluator → Reward function: computes rewards (evidence: `src/toyrlhf/exact.py`)
- REINFORCE → Sequence utilities: indexes prefixes (evidence: `src/toyrlhf/reinforce.py`)
- Actor-critic and PPO → Sequence utilities: indexes prefixes (evidence: `src/toyrlhf/actor_critic.py`)
- Neural policy → Actor-critic and PPO: uses GAE (evidence: `src/toyrlhf/neural.py`)
