How ydnyshhh/rlvr-gym works
1. Project shape
- RLVR-Gym is a Python library organized as a core framework plus several task-family modules.
- The package lives under src/rlvr_gym, with a small CLI, reusable core abstractions, family-specific generators, and documentation/artifact outputs.
2. Primary architectural boundary: core vs. families
- src/rlvr_gym/core contains the generic infrastructure that every task family uses.
- src/rlvr_gym/families contains the concrete environment families:
- deduction_grid
- graph_planning
- scheduling
- sokoban
- symbolic_transformation
- This is the main extension boundary: new task types are added as new family modules, not by modifying the runtime model.
3. Family abstraction and registry
- core/family.py defines the family abstraction: seeded sampling, environment creation, and config-driven task generation.
- registry.py is the lookup layer that exposes named families to the CLI and library consumers.
- The README’s get_family("...") usage reflects this indirection: callers work with family names, not concrete modules.
4. Task generation flow
- The generation pipeline is consistent across families:
- sample parameters from FamilyConfig and difficulty
- generate a latent world model
- derive the objective
- define observability/rendering
- attach verifier and reward settings
- optionally attach an oracle/solver
- build a runnable environment and exportable task spec
- This means the repo is centered on task synthesis, not on a single fixed simulator.
5. Runtime and interaction model
- core/runtime.py provides the lightweight environment loop with reset() / step() semantics via RLVREnv.
- The runtime records traces and exposes structured observations and valid actions.
- This is the main execution boundary between generated task definitions and agent interaction.
6. Verification and reward
- core/verifier.py encodes feasibility and correctness checks.
- core/reward.py defines how task outcomes are converted into reward signals.
- The architecture is “verification-first”: correctness is computed from the formal state/rules, not from heuristics or text matching.
7. Oracle and solution metadata
- core/oracle.py provides optional exact oracle support.
- Oracle output is used for rollout generation, label extraction, and proof-style solution metadata.
- Families that have tractable exact solutions can expose stronger supervision or benchmark labeling.
8. Export pipeline
- core/exporters.py is the conversion layer from a sampled task into downstream data formats.
- It supports multiple research workflows:
- task specs
- oracle rollouts
- SFT examples
- offline RL transitions
- oracle label views
- benchmark splits
- This is a key data boundary: generation/runtime on one side, consumable datasets and benchmarks on the other.
9. CLI as the operational entry point
- src/rlvr_gym/cli.py exposes commands like listing families, sampling tasks, and building benchmark splits.
- The CLI is a thin wrapper over the core/family/export APIs, so it does not define a separate execution model.
- The repository is intended to be run both as a library and as a command-line task factory.
10. Family implementations
- Each family module encodes its own latent problem structure and verification logic:
- graph_planning: weighted graph shortest-path style tasks
- scheduling: single-machine scheduling with deadlines and precedence constraints
- sokoban: solver-backed box-pushing/planning with deadlock diagnostics
- symbolic_transformation: symbolic rewriting and normalization
- deduction_grid: logic-grid deduction with exact table propagation
- These modules are where domain-specific state, action space, and task difficulty live.
11. Benchmark generation and frozen artifacts
- scripts/build_benchmark_artifacts.py likely generates the contents under docs/benchmark_artifacts.
- The benchmark artifact directories include spec files, split manifests, diversity summaries, and baseline results for each family.
- This shows a second major output path: beyond interactive environments, the repo produces frozen evaluation bundles.
12. Documentation as an architecture contract
- docs/architecture.md and the family-specific benchmark cards document intended capability coverage, verification guarantees, oracle semantics, metrics, and shortcut risks.
- These docs are part of the repo’s architecture because they define how each family should be interpreted and benchmarked.
13. Testing and validation
- tests/test_rlvr_gym.py is the main automated check for the package behavior.
- The test suite verifies that the library, CLI, and family registration/runtime path remain coherent.
14. Typical data flow to diagram
- A caller chooses a family name and seed/config.
- The registry resolves the family module.
- The family samples a task instance and builds an RLVREnv.
- The runtime emits observations/actions while verifiers and reward logic evaluate outcomes.
- Exporters transform the resulting task, trace, or oracle into benchmark or training artifacts.
15. Technology and operational context
- The project is pure Python, packaged with pyproject.toml, and managed with uv.
- It is optimized for research workflows around RLVR, offline RL, SFT, rejection sampling, and benchmark construction rather than for a production service stack.
Main entry points: family.py Family API · runtime.py Runtime · exporters.py Exporters · cli.py CLI
Components 18
Python package
Runtime library, CLI, and family modules under src/rlvr_gym
- CLI · Command-line entry point for listing, sampling, and building artifacts
src/rlvr_gym/cli.py - Registry · Resolves family names to concrete implementations
src/rlvr_gym/registry.py - Family API · Core family abstraction for seeded sampling and task construction
src/rlvr_gym/core/family.py - Runtime · Lightweight reset/step environment that records traces and observations
src/rlvr_gym/core/runtime.py - Verifier · Feasibility and solution checking against formal task rules
src/rlvr_gym/core/verifier.py - Reward · Converts task outcomes into reward values
src/rlvr_gym/core/reward.py - Oracle · Optional exact solver support for labels, rollouts, and metadata
src/rlvr_gym/core/oracle.py - Exporters · Converts sampled tasks and traces into research datasets and benchmark views
src/rlvr_gym/core/exporters.py - Types · Shared typed structures used across the core package
src/rlvr_gym/core/types.py - Utils · Shared utility functions used by core and families
src/rlvr_gym/core/utils.py
Task families
Concrete benchmark domains implemented as family modules
- Deduction grid · Logic-grid deduction tasks with exact table propagation
src/rlvr_gym/families/deduction_grid.py - Graph planning · Weighted graph shortest-path style tasks
src/rlvr_gym/families/graph_planning.py - Scheduling · Single-machine scheduling with deadlines and precedence constraints
src/rlvr_gym/families/scheduling.py - Sokoban · Solver-backed box pushing and planning tasks
src/rlvr_gym/families/sokoban.py - Symbolic transform · Symbolic rewriting and normalization tasks
src/rlvr_gym/families/symbolic_transformation.py
Docs & artifacts
Benchmark specs, cards, and frozen outputs
- Benchmark cards · Family-level benchmark guidance and evaluation contract
docs/benchmark_cards - Benchmark artifacts · Frozen specs, splits, diversity summaries, and baseline results
docs/benchmark_artifacts - Build script · Generates benchmark artifact bundles from family definitions
scripts/build_benchmark_artifacts.py
Connections 21
- CLI → Registry: resolve family · CLI looks up named families through the registry
- CLI → Exporters: build outputs · CLI can trigger task and benchmark exports
- Registry → Family API: binds · Registry exposes family implementations via the shared abstraction
- Family API → Runtime: creates env · Families construct runnable environments for agent interaction
- Family API → Verifier: attaches checks · Family generation includes formal validation logic
- Family API → Reward: attaches reward · Family generation includes task outcome scoring
- Family API → Oracle: optional solver · Some families expose exact oracle support
- Runtime → Verifier: evaluates · Runtime outcomes are checked against task rules
- Runtime → Reward: scores · Runtime transitions feed reward computation
- Oracle → Exporters: labels · Oracle outputs support rollouts, labels, and benchmark metadata
- Runtime → Exporters: traces · Recorded trajectories are transformed into export formats
- Family API → Exporters: task spec · Sampled tasks are converted into downstream artifacts
- Deduction grid → Family API: implements · Concrete family module follows the shared family contract
- Graph planning → Family API: implements · Concrete family module follows the shared family contract
- Scheduling → Family API: implements · Concrete family module follows the shared family contract
- Sokoban → Family API: implements · Concrete family module follows the shared family contract
- Symbolic transform → Family API: implements · Concrete family module follows the shared family contract
- Build script → Benchmark artifacts: generates · Build script produces frozen benchmark bundles
- Benchmark cards → Benchmark artifacts: documents · Benchmark cards describe the intended artifact and evaluation contract
- Types → Family API: shared models · Core abstractions reuse common typed structures
- Utils → Family API: supports · Utility helpers support core generation and runtime code