How ydnyshhh/rewardhack-gym works
1. What this repository is
rewardhack_gym is a research package for synthetic, verifiable environments where the official reward signal can diverge from the true objective. The architectural center is the pair of evaluators: a cheap official verifier and a stronger oracle, with the gap between them used to study reward hacking, post-training behavior, and mechanistic interpretability.
2. Top-level package shape
The code is organized into five main subsystems under src/rewardhack_gym:
- core: shared abstractions and configuration
- envs: concrete environment families
- runners: prompting / selection / RL-oriented wrappers
- analysis: metrics and interpretability-oriented summaries
- io: trace serialization
The CLI and package entrypoint sit on top of these modules and expose the same functionality to humans and scripts.
3. Core architecture boundary
rewardhack_gym.core defines the typed contract that every environment family follows: task objects, evaluation results, trajectories, configuration objects, and registry-backed lookup of environments by name. This is the main architectural seam: everything below it implements a specific task family, while everything above it consumes the shared result/trajectory model.
4. Environment families: the main domain layer
The envs subtree contains two families:
- envs.code: code-oriented tasks
- envs.math: math-oriented tasks
Each family contains both task definitions and a family module that groups the variants. The architecture is deliberately split by domain because the exploit surfaces, verifiers, and oracle checks differ substantially between code execution and mathematical reasoning.
5. Code environments
envs/code includes spec_overfit, patch_verification, interface_conformance, and schema_preservation. These are built around program artifacts, hidden regressions, contract violations, and semantic preservation issues. The file runtime.py indicates that code tasks need an execution harness for evaluating submitted outputs against tests or runtime behavior.
6. Math environments
envs/math contains symbolic_mismatch, piecewise_reasoning, reasoning_validity, and constraint_sensitive, plus shared helpers in common.py. These environments focus on branch-sensitive or constraint-sensitive reasoning, where official scoring can be weaker than the oracle. Compared with code, the math family is more about symbolic correctness, derivation validity, and domain-aware evaluation than runtime execution.
7. Evaluation flow
The core data flow is: create an environment → sample a task → produce or receive an output → evaluate that output under both official and oracle checks → emit a trajectory with labels and scores. The README’s task + official verifier + oracle + exploit surface primitive is reflected in the code layout and is the central diagrammatic path for the repository.
8. Exploitability profiles and configuration
The package supports controllable mismatch settings through profiles like low/medium/high/adversarial and knobs such as official coverage, hidden adversarial strength, parser strictness, process validation fraction, domain awareness, perturbation robustness, and public example diversity. These controls are important because they parameterize the environment’s exploit surface rather than just changing task difficulty.
9. Runners for policy-side experimentation
runners/prompting.py, runners/selection.py, and runners/rl.py are the policy-facing wrappers around the environments. They support prompting-style use, best-of-N or rejection-filter selection, and RL-oriented evaluation workflows. Architecturally, these modules sit between generated tasks and the environment evaluators, turning raw tasks into experiment protocols.
10. Analysis and interpretability preparation
analysis/metrics.py and analysis/mech_interp.py consume trajectories and produce summary statistics or mech-interp-oriented slices. The README emphasizes stable annotations such as scenario_id, canonical_exploit_class, exploit_strategy, and pass/fail breakdowns; those annotations are the bridge from environment execution to cohorting, false-pass mining, and downstream interpretability work.
11. Trace export and interchange
io/jsonl.py provides JSONL trace export, which is the main interchange format for downstream analysis. The repository treats traces as a first-class artifact: evaluation produces them, analysis consumes them, and export lets them move into external pipelines. The README also mentions optional Parquet support, but JSONL is the clearly visible built-in path in this tree.
12. CLI and package entrypoints
__main__.py and cli.py expose the repository through rewardhack-gym commands. The CLI surfaces the same boundaries as the library API: inspect environments, sample tasks, evaluate outputs, evaluate batches, compute stats, and export false passes. This makes the command-line interface a thin orchestration layer over the core/environment/analysis/io stack rather than a separate subsystem.
13. Execution model boundary
The repository explicitly treats code execution as trusted-local-only, not sandboxed. That is an important architectural boundary: the code runtime in envs/code/runtime.py supports evaluation, but it is not a security isolation layer. Any diagram should show execution as part of local research evaluation, not as a hardened remote service.
14. Testing and package validation
The tests cover CLI behavior, code and math environments, registry resolution, metrics and IO, profiles, and mechanistic-interpretability helpers. This mirrors the architecture closely: the test suite validates that the registry, environment families, trace export, and analytics all compose around the shared trajectory model.
Main entry points: base.py Base API · cli.py CLI · code/__init__.py Code family · math/__init__.py Math family
Components 24
Core
Shared task, trajectory, config, and registry contracts.
- Models · Typed task, trajectory, and evaluation result objects.
src/rewardhack_gym/core/models.py - Config · Profiles and knobs controlling exploitability and environment behavior.
src/rewardhack_gym/core/config.py - Registry · Name-based environment registration and resolution.
src/rewardhack_gym/core/registry.py - Base API · Shared base abstractions that environments implement.
src/rewardhack_gym/core/base.py
Code envs
Program-oriented task families and runtime harness.
- Code family · Code-task variants grouped as one domain subsystem.
src/rewardhack_gym/envs/code/__init__.py - Runtime · Local execution layer used to evaluate submitted code artifacts.
src/rewardhack_gym/envs/code/runtime.py - Spec overfit · Tasks around fitting specifications too narrowly.
src/rewardhack_gym/envs/code/spec_overfit.py - Patch verify · Patch and regression-style verification tasks.
src/rewardhack_gym/envs/code/patch_verification.py - Interface · API and contract conformance tasks.
src/rewardhack_gym/envs/code/interface_conformance.py - Schema keep · Tasks about preserving structure and schema semantics.
src/rewardhack_gym/envs/code/schema_preservation.py
Math envs
Reasoning-oriented task families and shared helpers.
- Math family · Reasoning-task variants grouped as one domain subsystem.
src/rewardhack_gym/envs/math/__init__.py - Math common · Shared helpers and annotations for math environments.
src/rewardhack_gym/envs/math/common.py - Symbolic mismatch · Tasks where symbolic structure and official scoring diverge.
src/rewardhack_gym/envs/math/symbolic_mismatch.py - Piecewise · Branch-sensitive reasoning tasks with case-dependent validity.
src/rewardhack_gym/envs/math/piecewise_reasoning.py - Validity · Tasks centered on derivation and reasoning correctness.
src/rewardhack_gym/envs/math/reasoning_validity.py - Constraints · Constraint-sensitive tasks where domain awareness matters.
src/rewardhack_gym/envs/math/constraint_sensitive.py
Runners
Policy-side wrappers for prompting, selection, and RL workflows.
- Prompting · Prompt-based task execution and evaluation wrapper.
src/rewardhack_gym/runners/prompting.py - Selection · Best-of-N and rejection-filter style selection workflows.
src/rewardhack_gym/runners/selection.py - RL runner · RL-oriented experiment wrapper over environment evaluation.
src/rewardhack_gym/runners/rl.py
Analysis
Metrics and mech-interp oriented summaries over trajectories.
- Metrics · Summary statistics, breakdowns, and cohort metrics over trajectories.
src/rewardhack_gym/analysis/metrics.py - Mech interp · Mechanistic-interpretability-oriented slicing and annotations.
src/rewardhack_gym/analysis/mech_interp.py
Trace IO
Trace serialization and interchange.
- JSONL export · Trace export and interchange format for downstream pipelines.
src/rewardhack_gym/io/jsonl.py
Entrypoints
CLI and package entrypoints.
- CLI · User-facing command-line orchestration over the package.
src/rewardhack_gym/cli.py - Entrypoint · Python package entrypoint for command execution.
src/rewardhack_gym/__main__.py
Connections 28
- Registry → Base API: registers · Registry resolves concrete environment implementations through shared contracts.
- Base API → Models: uses · Base environment APIs operate on typed tasks, trajectories, and results.
- Config → Base API: configures · Exploitability and environment settings parameterize the base evaluation contract.
- Base API → Code family: specializes · Code environments implement the shared environment interface.
- Base API → Math family: specializes · Math environments implement the shared environment interface.
- Code family → Runtime: evaluates via · Code tasks use a local execution harness for verification.
- Code family → Spec overfit: includes · One code-task variant in the family.
- Code family → Patch verify: includes · One code-task variant in the family.
- Code family → Interface: includes · One code-task variant in the family.
- Code family → Schema keep: includes · One code-task variant in the family.
- Math family → Math common: shares · Math environments reuse common helpers and annotations.
- Math family → Symbolic mismatch: includes · One math-task variant in the family.
- Math family → Piecewise: includes · One math-task variant in the family.
- Math family → Validity: includes · One math-task variant in the family.
- Math family → Constraints: includes · One math-task variant in the family.
- Prompting → Base API: drives · Prompting workflows create and evaluate tasks through the core API.
- Selection → Base API: filters · Selection workflows operate on environment outputs and scores.
- RL runner → Base API: trains on · RL workflows consume the same task and trajectory model.
- Metrics → Models: consumes · Metrics summarize trajectories and pass/fail labels.
- Mech interp → Models: slices · Mech-interp helpers build cohorts from trajectory annotations.
- JSONL export → Models: serializes · Trace export writes the shared trajectory representation.
- CLI → Registry: uses · CLI resolves available environments through the registry.
- CLI → Prompting: exposes · Command-line entrypoints surface prompting workflows.
- CLI → Selection: exposes · Command-line entrypoints surface selection workflows.
- CLI → RL runner: exposes · Command-line entrypoints surface RL workflows.
- CLI → Metrics: exposes · Command-line entrypoints surface summary statistics.
- CLI → JSONL export: exposes · Command-line entrypoints surface trace export.
- Entrypoint → CLI: invokes · Module entrypoint dispatches to the CLI.