# ydnyshhh/rewardhack-gym architecture

> Architecture diagram of the GitHub repository ydnyshhh/rewardhack-gym, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ydnyshhh/rewardhack-gym
- Repository: https://github.com/ydnyshhh/rewardhack-gym

## Overview

1. **What this repository is**
   `rewardhack_gym` is a research package for synthetic, verifiable environments where the official reward signal can diverge from the true objective. The architectural center is the pair of evaluators: a cheap **official verifier** and a stronger **oracle**, with the gap between them used to study reward hacking, post-training behavior, and mechanistic interpretability.

2. **Top-level package shape**
   The code is organized into five main subsystems under `src/rewardhack_gym`:
   - `core`: shared abstractions and configuration
   - `envs`: concrete environment families
   - `runners`: prompting / selection / RL-oriented wrappers
   - `analysis`: metrics and interpretability-oriented summaries
   - `io`: trace serialization

   The CLI and package entrypoint sit on top of these modules and expose the same functionality to humans and scripts.

3. **Core architecture boundary**
   `rewardhack_gym.core` defines the typed contract that every environment family follows: task objects, evaluation results, trajectories, configuration objects, and registry-backed lookup of environments by name. This is the main architectural seam: everything below it implements a specific task family, while everything above it consumes the shared result/trajectory model.

4. **Environment families: the main domain layer**
   The `envs` subtree contains two families:
   - `envs.code`: code-oriented tasks
   - `envs.math`: math-oriented tasks

   Each family contains both task definitions and a family module that groups the variants. The architecture is deliberately split by domain because the exploit surfaces, verifiers, and oracle checks differ substantially between code execution and mathematical reasoning.

5. **Code environments**
   `envs/code` includes `spec_overfit`, `patch_verification`, `interface_conformance`, and `schema_preservation`. These are built around program artifacts, hidden regressions, contract violations, and semantic preservation issues. The file `runtime.py` indicates that code tasks need an execution harness for evaluating submitted outputs against tests or runtime behavior.

6. **Math environments**
   `envs/math` contains `symbolic_mismatch`, `piecewise_reasoning`, `reasoning_validity`, and `constraint_sensitive`, plus shared helpers in `common.py`. These environments focus on branch-sensitive or constraint-sensitive reasoning, where official scoring can be weaker than the oracle. Compared with code, the math family is more about symbolic correctness, derivation validity, and domain-aware evaluation than runtime execution.

7. **Evaluation flow**
   The core data flow is: create an environment → sample a task → produce or receive an output → evaluate that output under both official and oracle checks → emit a trajectory with labels and scores. The README’s `task + official verifier + oracle + exploit surface` primitive is reflected in the code layout and is the central diagrammatic path for the repository.

8. **Exploitability profiles and configuration**
   The package supports controllable mismatch settings through profiles like low/medium/high/adversarial and knobs such as official coverage, hidden adversarial strength, parser strictness, process validation fraction, domain awareness, perturbation robustness, and public example diversity. These controls are important because they parameterize the environment’s exploit surface rather than just changing task difficulty.

9. **Runners for policy-side experimentation**
   `runners/prompting.py`, `runners/selection.py`, and `runners/rl.py` are the policy-facing wrappers around the environments. They support prompting-style use, best-of-N or rejection-filter selection, and RL-oriented evaluation workflows. Architecturally, these modules sit between generated tasks and the environment evaluators, turning raw tasks into experiment protocols.

10. **Analysis and interpretability preparation**
    `analysis/metrics.py` and `analysis/mech_interp.py` consume trajectories and produce summary statistics or mech-interp-oriented slices. The README emphasizes stable annotations such as `scenario_id`, `canonical_exploit_class`, `exploit_strategy`, and pass/fail breakdowns; those annotations are the bridge from environment execution to cohorting, false-pass mining, and downstream interpretability work.

11. **Trace export and interchange**
    `io/jsonl.py` provides JSONL trace export, which is the main interchange format for downstream analysis. The repository treats traces as a first-class artifact: evaluation produces them, analysis consumes them, and export lets them move into external pipelines. The README also mentions optional Parquet support, but JSONL is the clearly visible built-in path in this tree.

12. **CLI and package entrypoints**
    `__main__.py` and `cli.py` expose the repository through `rewardhack-gym` commands. The CLI surfaces the same boundaries as the library API: inspect environments, sample tasks, evaluate outputs, evaluate batches, compute stats, and export false passes. This makes the command-line interface a thin orchestration layer over the core/environment/analysis/io stack rather than a separate subsystem.

13. **Execution model boundary**
    The repository explicitly treats code execution as trusted-local-only, not sandboxed. That is an important architectural boundary: the code runtime in `envs/code/runtime.py` supports evaluation, but it is not a security isolation layer. Any diagram should show execution as part of local research evaluation, not as a hardened remote service.

14. **Testing and package validation**
    The tests cover CLI behavior, code and math environments, registry resolution, metrics and IO, profiles, and mechanistic-interpretability helpers. This mirrors the architecture closely: the test suite validates that the registry, environment families, trace export, and analytics all compose around the shared trajectory model.

Main entry points: [`src/rewardhack_gym/core/base.py`](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/core/base.py) (Base API), [`src/rewardhack_gym/cli.py`](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/cli.py) (CLI), [`src/rewardhack_gym/envs/code/__init__.py`](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/__init__.py) (Code family), [`src/rewardhack_gym/envs/math/__init__.py`](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/__init__.py) (Math family)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_core["Core"]
  node_node_core_models["Models<br/>trajectory/results<br/>[models.py]"]
  node_node_core_config["Config<br/>mismatch settings<br/>[config.py]"]
  node_node_core_registry["Registry<br/>env lookup<br/>[registry.py]"]
  node_node_core_base["Base API<br/>contracts<br/>[base.py]"]
end

subgraph group_group_envs_code["Code envs"]
  node_node_code_family["Code family<br/>code env group<br/>[__init__.py]"]
  node_node_code_runtime["Runtime<br/>execution harness<br/>[runtime.py]"]
  node_node_code_spec["Spec overfit<br/>code task<br/>[spec_overfit.py]"]
  node_node_code_patch["Patch verify<br/>code task"]
  node_node_code_iface["Interface<br/>code task"]
  node_node_code_schema["Schema keep<br/>code task"]
end

subgraph group_group_envs_math["Math envs"]
  node_node_math_family["Math family<br/>math env group<br/>[__init__.py]"]
  node_node_math_common["Math common<br/>shared helpers<br/>[common.py]"]
  node_node_math_symbolic["Symbolic mismatch<br/>math task"]
  node_node_math_piecewise["Piecewise<br/>math task"]
  node_node_math_validity["Validity<br/>math task"]
  node_node_math_constraint["Constraints<br/>math task"]
end

subgraph group_group_runners["Runners"]
  node_node_runner_prompt["Prompting<br/>policy runner<br/>[prompting.py]"]
  node_node_runner_select["Selection<br/>policy runner<br/>[selection.py]"]
  node_node_runner_rl["RL runner<br/>policy runner<br/>[rl.py]"]
end

subgraph group_group_analysis["Analysis"]
  node_node_analysis_metrics["Metrics<br/>trajectory stats<br/>[metrics.py]"]
  node_node_analysis_mech["Mech interp<br/>trajectory slices<br/>[mech_interp.py]"]
end

subgraph group_group_io["Trace IO"]
  node_node_io_jsonl["JSONL export<br/>trace serialisation<br/>[jsonl.py]"]
end

subgraph group_group_entry["Entrypoints"]
  node_node_cli["CLI<br/>command layer<br/>[cli.py]"]
  node_node_main["Entrypoint<br/>module entry<br/>[__main__.py]"]
end

node_node_core_registry -->|"registers"| node_node_core_base
node_node_core_base -->|"uses"| node_node_core_models
node_node_core_config -->|"configures"| node_node_core_base
node_node_core_base -->|"specializes"| node_node_code_family
node_node_core_base -->|"specializes"| node_node_math_family
node_node_code_family -->|"evaluates via"| node_node_code_runtime
node_node_code_family -->|"includes"| node_node_code_spec
node_node_code_family -->|"includes"| node_node_code_patch
node_node_code_family -->|"includes"| node_node_code_iface
node_node_code_family -->|"includes"| node_node_code_schema
node_node_math_family -->|"shares"| node_node_math_common
node_node_math_family -->|"includes"| node_node_math_symbolic
node_node_math_family -->|"includes"| node_node_math_piecewise
node_node_math_family -->|"includes"| node_node_math_validity
node_node_math_family -->|"includes"| node_node_math_constraint
node_node_runner_prompt -->|"drives"| node_node_core_base
node_node_runner_select -->|"filters"| node_node_core_base
node_node_runner_rl -->|"trains on"| node_node_core_base
node_node_analysis_metrics -->|"consumes"| node_node_core_models
node_node_analysis_mech -->|"slices"| node_node_core_models
node_node_io_jsonl -->|"serializes"| node_node_core_models
node_node_cli -->|"uses"| node_node_core_registry
node_node_cli -->|"exposes"| node_node_runner_prompt
node_node_cli -->|"exposes"| node_node_runner_select
node_node_cli -->|"exposes"| node_node_runner_rl
node_node_cli -->|"exposes"| node_node_analysis_metrics
node_node_cli -->|"exposes"| node_node_io_jsonl
node_node_main -->|"invokes"| node_node_cli

click node_node_core_models "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/core/models.py"
click node_node_core_config "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/core/config.py"
click node_node_core_registry "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/core/registry.py"
click node_node_core_base "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/core/base.py"
click node_node_code_family "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/__init__.py"
click node_node_code_runtime "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/runtime.py"
click node_node_code_spec "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/spec_overfit.py"
click node_node_code_patch "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/patch_verification.py"
click node_node_code_iface "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/interface_conformance.py"
click node_node_code_schema "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/schema_preservation.py"
click node_node_math_family "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/__init__.py"
click node_node_math_common "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/common.py"
click node_node_math_symbolic "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/symbolic_mismatch.py"
click node_node_math_piecewise "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/piecewise_reasoning.py"
click node_node_math_validity "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/reasoning_validity.py"
click node_node_math_constraint "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/constraint_sensitive.py"
click node_node_runner_prompt "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/runners/prompting.py"
click node_node_runner_select "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/runners/selection.py"
click node_node_runner_rl "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/runners/rl.py"
click node_node_analysis_metrics "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/analysis/metrics.py"
click node_node_analysis_mech "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/analysis/mech_interp.py"
click node_node_io_jsonl "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/io/jsonl.py"
click node_node_cli "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/cli.py"
click node_node_main "https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/__main__.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_core_models,node_node_core_config,node_node_core_registry,node_node_core_base,node_node_cli,node_node_main toneBlue
class node_node_code_family,node_node_code_runtime,node_node_code_spec,node_node_code_patch,node_node_code_iface,node_node_code_schema toneAmber
class node_node_math_family,node_node_math_common,node_node_math_symbolic,node_node_math_piecewise,node_node_math_validity,node_node_math_constraint toneMint
class node_node_runner_prompt,node_node_runner_select,node_node_runner_rl toneRose
class node_node_analysis_metrics,node_node_analysis_mech toneIndigo
class node_node_io_jsonl toneTeal
```

## Components

### Core

Shared task, trajectory, config, and registry contracts.

- [Models](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/core/models.py): trajectory/results · Typed task, trajectory, and evaluation result objects. · `src/rewardhack_gym/core/models.py`
- [Config](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/core/config.py): mismatch settings · Profiles and knobs controlling exploitability and environment behavior. · `src/rewardhack_gym/core/config.py`
- [Registry](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/core/registry.py): env lookup · Name-based environment registration and resolution. · `src/rewardhack_gym/core/registry.py`
- [Base API](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/core/base.py): contracts · Shared base abstractions that environments implement. · `src/rewardhack_gym/core/base.py`

### Code envs

Program-oriented task families and runtime harness.

- [Code family](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/__init__.py): code env group · Code-task variants grouped as one domain subsystem. · `src/rewardhack_gym/envs/code/__init__.py`
- [Runtime](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/runtime.py): execution harness · Local execution layer used to evaluate submitted code artifacts. · `src/rewardhack_gym/envs/code/runtime.py`
- [Spec overfit](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/spec_overfit.py): code task · Tasks around fitting specifications too narrowly. · `src/rewardhack_gym/envs/code/spec_overfit.py`
- [Patch verify](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/patch_verification.py): code task · Patch and regression-style verification tasks. · `src/rewardhack_gym/envs/code/patch_verification.py`
- [Interface](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/interface_conformance.py): code task · API and contract conformance tasks. · `src/rewardhack_gym/envs/code/interface_conformance.py`
- [Schema keep](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/code/schema_preservation.py): code task · Tasks about preserving structure and schema semantics. · `src/rewardhack_gym/envs/code/schema_preservation.py`

### Math envs

Reasoning-oriented task families and shared helpers.

- [Math family](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/__init__.py): math env group · Reasoning-task variants grouped as one domain subsystem. · `src/rewardhack_gym/envs/math/__init__.py`
- [Math common](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/common.py): shared helpers · Shared helpers and annotations for math environments. · `src/rewardhack_gym/envs/math/common.py`
- [Symbolic mismatch](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/symbolic_mismatch.py): math task · Tasks where symbolic structure and official scoring diverge. · `src/rewardhack_gym/envs/math/symbolic_mismatch.py`
- [Piecewise](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/piecewise_reasoning.py): math task · Branch-sensitive reasoning tasks with case-dependent validity. · `src/rewardhack_gym/envs/math/piecewise_reasoning.py`
- [Validity](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/reasoning_validity.py): math task · Tasks centered on derivation and reasoning correctness. · `src/rewardhack_gym/envs/math/reasoning_validity.py`
- [Constraints](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/envs/math/constraint_sensitive.py): math task · Constraint-sensitive tasks where domain awareness matters. · `src/rewardhack_gym/envs/math/constraint_sensitive.py`

### Runners

Policy-side wrappers for prompting, selection, and RL workflows.

- [Prompting](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/runners/prompting.py): policy runner · Prompt-based task execution and evaluation wrapper. · `src/rewardhack_gym/runners/prompting.py`
- [Selection](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/runners/selection.py): policy runner · Best-of-N and rejection-filter style selection workflows. · `src/rewardhack_gym/runners/selection.py`
- [RL runner](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/runners/rl.py): policy runner · RL-oriented experiment wrapper over environment evaluation. · `src/rewardhack_gym/runners/rl.py`

### Analysis

Metrics and mech-interp oriented summaries over trajectories.

- [Metrics](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/analysis/metrics.py): trajectory stats · Summary statistics, breakdowns, and cohort metrics over trajectories. · `src/rewardhack_gym/analysis/metrics.py`
- [Mech interp](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/analysis/mech_interp.py): trajectory slices · Mechanistic-interpretability-oriented slicing and annotations. · `src/rewardhack_gym/analysis/mech_interp.py`

### Trace IO

Trace serialization and interchange.

- [JSONL export](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/io/jsonl.py): trace serialisation · Trace export and interchange format for downstream pipelines. · `src/rewardhack_gym/io/jsonl.py`

### Entrypoints

CLI and package entrypoints.

- [CLI](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/cli.py): command layer · User-facing command-line orchestration over the package. · `src/rewardhack_gym/cli.py`
- [Entrypoint](https://github.com/ydnyshhh/rewardhack-gym/blob/main/src/rewardhack_gym/__main__.py): module entry · Python package entrypoint for command execution. · `src/rewardhack_gym/__main__.py`

## Connections

- Registry → Base API: registers · Registry resolves concrete environment implementations through shared contracts.
- Base API → Models: uses · Base environment APIs operate on typed tasks, trajectories, and results.
- Config → Base API: configures · Exploitability and environment settings parameterize the base evaluation contract.
- Base API → Code family: specializes · Code environments implement the shared environment interface.
- Base API → Math family: specializes · Math environments implement the shared environment interface.
- Code family → Runtime: evaluates via · Code tasks use a local execution harness for verification.
- Code family → Spec overfit: includes · One code-task variant in the family.
- Code family → Patch verify: includes · One code-task variant in the family.
- Code family → Interface: includes · One code-task variant in the family.
- Code family → Schema keep: includes · One code-task variant in the family.
- Math family → Math common: shares · Math environments reuse common helpers and annotations.
- Math family → Symbolic mismatch: includes · One math-task variant in the family.
- Math family → Piecewise: includes · One math-task variant in the family.
- Math family → Validity: includes · One math-task variant in the family.
- Math family → Constraints: includes · One math-task variant in the family.
- Prompting → Base API: drives · Prompting workflows create and evaluate tasks through the core API.
- Selection → Base API: filters · Selection workflows operate on environment outputs and scores.
- RL runner → Base API: trains on · RL workflows consume the same task and trajectory model.
- Metrics → Models: consumes · Metrics summarize trajectories and pass/fail labels.
- Mech interp → Models: slices · Mech-interp helpers build cohorts from trajectory annotations.
- JSONL export → Models: serializes · Trace export writes the shared trajectory representation.
- CLI → Registry: uses · CLI resolves available environments through the registry.
- CLI → Prompting: exposes · Command-line entrypoints surface prompting workflows.
- CLI → Selection: exposes · Command-line entrypoints surface selection workflows.
- CLI → RL runner: exposes · Command-line entrypoints surface RL workflows.
- CLI → Metrics: exposes · Command-line entrypoints surface summary statistics.
- CLI → JSONL export: exposes · Command-line entrypoints surface trace export.
- Entrypoint → CLI: invokes · Module entrypoint dispatches to the CLI.
