How ydnyshhh/synthetic-workspace-gym works
1. Synthetic Workspace Gym is a Python package for generating, running, and evaluating synthetic agent environments. The core unit is an environment instance with a visible workspace, hidden evaluator assets, a manifest, and a reference solution. The system is designed for research on agent behavior in executable workspaces rather than prompt-only tasks.
2. The repository is organized around four main environment families: tabular, script_repair, pipeline, and retrieval_workspace. Each family has its own generator and evaluator implementation, plus scenario modules that define concrete task templates and difficulty scaling. This is the main domain-specific boundary in the architecture.
3. The generation subsystem creates concrete environment instances from an EnvironmentSpec. Generators sample or accept scenario/difficulty/seed inputs, construct visible files and hidden assets, and emit an EnvironmentManifest. The manifest includes the evaluator entrypoint, metadata, and reference solution, which later drive execution and trusted scoring.
4. The runtime subsystem loads a generated environment, copies the visible workspace into a fresh scratch area, and executes an episode through a tool-based loop. Its main components are the environment loader, tool executor, policy layer, and EpisodeRunner. This is the boundary where agent actions are converted into file edits and shell/Python operations.
5. The agent subsystem provides baseline policies rather than full learning infrastructure. scripted is a minimal heuristic baseline, and react.py contains a compatibility alias for older ReAct-style use. The more privileged heuristic baseline mentioned in the README is effectively a replay/validation path based on the manifest’s stored reference solution, not a true reasoning agent.
6. The evaluator subsystem is the trusted scoring layer. Evaluators are resolved dynamically from the manifest’s evaluator_entrypoint, so the generated environment determines which evaluator class runs. Evaluators operate outside the writable visible workspace boundary and inspect the final workspace plus hidden assets to produce an EvaluatorResult with success, score, subscores, failure labels, diagnostics, and runtime duration.
7. The schemas package is the shared contract layer for the whole system. It defines typed dataclass-based structures such as EnvironmentSpec, EnvironmentManifest, EvaluatorResult, and step-level trajectory records. This schema boundary keeps generators, runtime, evaluators, and analysis aligned without tight coupling.
8. The analysis subsystem handles artifact export and benchmarking. It snapshots manifests, trajectories, diffs, final workspaces, and evaluator outputs into episode directories, then aggregates results across runs. The benchmark path normalizes per-episode rows and produces grouped summaries by family, difficulty, scenario, and family-specific metadata such as bug scope or retrieval characteristics.
9. The CLI is the orchestration entrypoint for the whole repo. swg generate builds environments, swg run executes an agent episode, swg evaluate runs trusted scoring on a workspace, and swg benchmark sweeps a baseline over many environments. The CLI is the user-facing composition layer over generation, runtime, and analysis.
10. The runtime model is intentionally local and partially isolated rather than a hardened sandbox. Episodes run in a scratch copy of visible/, while hidden/ remains outside the agent’s writable surface. The tool layer enforces denylist-style guardrails against parent traversal, absolute paths, inline env injection, inline Python via shell, and common network commands, so the architecture assumes local integrity but not hostile-code security.
11. Data flow is: spec or scenario selection → generated environment directory with manifest and visible/hidden files → episode runner loads the manifest and workspace → agent performs tool actions in the scratch workspace → final workspace is evaluated by the hidden evaluator → artifacts and benchmark rows are exported. That pipeline is the key end-to-end path to draw in an architecture diagram.
12. The test suite mirrors these boundaries with coverage for schemas, generators, evaluators, runtime, CLI, benchmarking, and end-to-end behavior. That indicates the repository is structured as research infrastructure with strong contract checks between generation, execution, and trusted evaluation rather than as a monolithic application.
Main entry points: generators/base.py Generators · evaluators/base.py Evaluators · runner.py Runtime · cli.py CLI
Components 18
Core system
Contracts, generation, execution, and scoring for synthetic workspaces.
- Schemas · Shared typed structures for specs, manifests, trajectories, and evaluator outputs.
src/synthetic_workspace_gym/schemas/common.py - Generators · Creates environment directories, manifests, visible files, and hidden evaluator assets.
src/synthetic_workspace_gym/generators/base.py - Gen registry · Resolves generator implementations by family or scenario.
src/synthetic_workspace_gym/generators/registry.py - Runtime · Loads generated environments and runs tool-based episodes in a scratch workspace.
src/synthetic_workspace_gym/runtime/runner.py - Loader · Copies the visible workspace into an isolated scratch area and prepares episode state.
src/synthetic_workspace_gym/runtime/environment.py - Tools · Executes file and shell actions with local guardrails.
src/synthetic_workspace_gym/runtime/tools.py - Policy · Connects the runner to baseline policies and action selection.
src/synthetic_workspace_gym/runtime/policy.py - Evaluators · Resolves and runs hidden evaluators against the final workspace and assets.
src/synthetic_workspace_gym/evaluators/base.py - Eval registry · Dynamic evaluator lookup from manifest entrypoints and family registration.
src/synthetic_workspace_gym/evaluators/registry.py - Utils · Shared I/O, path, and scratch-space helpers used across the repo.
src/synthetic_workspace_gym/utils/scratch.py
Environment families
Task-family implementations for concrete synthetic environments.
- Tabular · Tabular environment generation and evaluation family.
src/synthetic_workspace_gym/generators/tabular.py - Script repair · Script-fixing environment family with its own scenarios and evaluator.
src/synthetic_workspace_gym/generators/script_repair.py - Pipeline · Pipeline completion environment family and scenarios.
src/synthetic_workspace_gym/generators/pipeline_completion.py - Retrieval workspace · Retrieval-oriented workspace family with hidden assets and evaluation.
src/synthetic_workspace_gym/generators/retrieval_workspace.py
Operations
User-facing entrypoints and result processing.
- CLI · Command-line orchestration for generation, execution, evaluation, and benchmarking.
src/synthetic_workspace_gym/cli.py - Agents · Minimal scripted and compatibility baselines used for episodes and benchmarks.
src/synthetic_workspace_gym/agents/base.py - Analysis · Exports episode artifacts and aggregates benchmark results.
src/synthetic_workspace_gym/analysis/artifacts.py - Benchmarking · Normalizes run rows and produces grouped benchmark summaries.
src/synthetic_workspace_gym/analysis/benchmarking.py
Connections 26
- CLI → Generators: generate · Builds environment instances from specs and scenario choices.
- CLI → Runtime: run · Executes an agent episode in a generated environment.
- CLI → Evaluators: evaluate · Runs trusted scoring on a workspace.
- CLI → Analysis: benchmark · Exports artifacts and sweeps baselines across environments.
- Generators → Schemas: emit manifest · Produces EnvironmentSpec and EnvironmentManifest data.
- Generators → Gen registry: resolve family · Selects family-specific generator implementations.
- Generators → Tabular: family impl · Generates tabular environments and scenarios.
- Generators → Script repair: family impl · Generates script repair environments and scenarios.
- Generators → Pipeline: family impl · Generates pipeline environments and scenarios.
- Generators → Retrieval workspace: family impl · Generates retrieval workspace environments and scenarios.
- Runtime → Loader: load workspace · Prepares scratch copies of visible files for an episode.
- Runtime → Tools: actions · Applies file edits and shell/python operations.
- Runtime → Policy: policy loop · Queries the selected baseline policy for next actions.
- Policy → Agents: baseline · Uses scripted and compatibility agent implementations.
- Runtime → Schemas: trajectory · Produces step-level records and episode outputs.
- Evaluators → Eval registry: resolve · Loads the manifest-specified evaluator entrypoint.
- Evaluators → Schemas: result · Produces EvaluatorResult and subscores.
- Evaluators → Tabular: family eval · Tabular scoring logic.
- Evaluators → Script repair: family eval · Script repair scoring logic.
- Evaluators → Pipeline: family eval · Pipeline scoring logic.
- Evaluators → Retrieval workspace: family eval · Retrieval workspace scoring logic.
- Analysis → Schemas: consume · Reads manifests, trajectories, diffs, and evaluator outputs.
- Analysis → Benchmarking: aggregate · Builds grouped summaries across runs.
- Utils → Generators: helpers · Supports path handling and workspace creation.
- Utils → Runtime: helpers · Supports scratch-space and file operations.
- Utils → Analysis: helpers · Supports artifact I/O and path management.