# ydnyshhh/synthetic-workspace-gym architecture

> Architecture diagram of the GitHub repository ydnyshhh/synthetic-workspace-gym, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ydnyshhh/synthetic-workspace-gym
- Repository: https://github.com/ydnyshhh/synthetic-workspace-gym

## Overview

1. Synthetic Workspace Gym is a Python package for generating, running, and evaluating synthetic agent environments. The core unit is an environment instance with a visible workspace, hidden evaluator assets, a manifest, and a reference solution. The system is designed for research on agent behavior in executable workspaces rather than prompt-only tasks.

2. The repository is organized around four main environment families: `tabular`, `script_repair`, `pipeline`, and `retrieval_workspace`. Each family has its own generator and evaluator implementation, plus scenario modules that define concrete task templates and difficulty scaling. This is the main domain-specific boundary in the architecture.

3. The generation subsystem creates concrete environment instances from an `EnvironmentSpec`. Generators sample or accept scenario/difficulty/seed inputs, construct visible files and hidden assets, and emit an `EnvironmentManifest`. The manifest includes the evaluator entrypoint, metadata, and reference solution, which later drive execution and trusted scoring.

4. The runtime subsystem loads a generated environment, copies the visible workspace into a fresh scratch area, and executes an episode through a tool-based loop. Its main components are the environment loader, tool executor, policy layer, and `EpisodeRunner`. This is the boundary where agent actions are converted into file edits and shell/Python operations.

5. The agent subsystem provides baseline policies rather than full learning infrastructure. `scripted` is a minimal heuristic baseline, and `react.py` contains a compatibility alias for older ReAct-style use. The more privileged `heuristic` baseline mentioned in the README is effectively a replay/validation path based on the manifest’s stored reference solution, not a true reasoning agent.

6. The evaluator subsystem is the trusted scoring layer. Evaluators are resolved dynamically from the manifest’s `evaluator_entrypoint`, so the generated environment determines which evaluator class runs. Evaluators operate outside the writable visible workspace boundary and inspect the final workspace plus hidden assets to produce an `EvaluatorResult` with success, score, subscores, failure labels, diagnostics, and runtime duration.

7. The `schemas` package is the shared contract layer for the whole system. It defines typed dataclass-based structures such as `EnvironmentSpec`, `EnvironmentManifest`, `EvaluatorResult`, and step-level trajectory records. This schema boundary keeps generators, runtime, evaluators, and analysis aligned without tight coupling.

8. The analysis subsystem handles artifact export and benchmarking. It snapshots manifests, trajectories, diffs, final workspaces, and evaluator outputs into episode directories, then aggregates results across runs. The benchmark path normalizes per-episode rows and produces grouped summaries by family, difficulty, scenario, and family-specific metadata such as bug scope or retrieval characteristics.

9. The CLI is the orchestration entrypoint for the whole repo. `swg generate` builds environments, `swg run` executes an agent episode, `swg evaluate` runs trusted scoring on a workspace, and `swg benchmark` sweeps a baseline over many environments. The CLI is the user-facing composition layer over generation, runtime, and analysis.

10. The runtime model is intentionally local and partially isolated rather than a hardened sandbox. Episodes run in a scratch copy of `visible/`, while `hidden/` remains outside the agent’s writable surface. The tool layer enforces denylist-style guardrails against parent traversal, absolute paths, inline env injection, inline Python via shell, and common network commands, so the architecture assumes local integrity but not hostile-code security.

11. Data flow is: spec or scenario selection → generated environment directory with manifest and visible/hidden files → episode runner loads the manifest and workspace → agent performs tool actions in the scratch workspace → final workspace is evaluated by the hidden evaluator → artifacts and benchmark rows are exported. That pipeline is the key end-to-end path to draw in an architecture diagram.

12. The test suite mirrors these boundaries with coverage for schemas, generators, evaluators, runtime, CLI, benchmarking, and end-to-end behavior. That indicates the repository is structured as research infrastructure with strong contract checks between generation, execution, and trusted evaluation rather than as a monolithic application.

Main entry points: [`src/synthetic_workspace_gym/generators/base.py`](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/base.py) (Generators), [`src/synthetic_workspace_gym/evaluators/base.py`](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/evaluators/base.py) (Evaluators), [`src/synthetic_workspace_gym/runtime/runner.py`](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/runtime/runner.py) (Runtime), [`src/synthetic_workspace_gym/cli.py`](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/cli.py) (CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core system"]
  node_schemas["Schemas<br/>contracts<br/>[common.py]"]
  node_generators["Generators<br/>build envs<br/>[base.py]"]
  node_generator_registry{{"Gen registry<br/>resolver<br/>[registry.py]"}}
  node_runtime["Runtime<br/>episode exec<br/>[runner.py]"]
  node_environment_loader["Loader<br/>workspace load<br/>[environment.py]"]
  node_tool_executor["Tools<br/>action layer<br/>[tools.py]"]
  node_policy_layer["Policy<br/>agent bridge<br/>[policy.py]"]
  node_evaluators["Evaluators<br/>trusted scoring<br/>[base.py]"]
  node_evaluator_registry{{"Eval registry<br/>resolver<br/>[registry.py]"}}
  node_utils["Utils<br/>support<br/>[scratch.py]"]
end

subgraph group_domain["Environment families"]
  node_tabular["Tabular<br/>family<br/>[tabular.py]"]
  node_script_repair["Script repair<br/>family<br/>[script_repair.py]"]
  node_pipeline["Pipeline<br/>family"]
  node_retrieval_workspace["Retrieval workspace<br/>family"]
end

subgraph group_ops["Operations"]
  node_cli["CLI<br/>entrypoint<br/>[cli.py]"]
  node_agents["Agents<br/>baselines<br/>[base.py]"]
  node_analysis["Analysis<br/>artifacts<br/>[artifacts.py]"]
  node_benchmarking["Benchmarking<br/>summaries<br/>[benchmarking.py]"]
end

node_cli -->|"generate"| node_generators
node_cli -->|"run"| node_runtime
node_cli -->|"evaluate"| node_evaluators
node_cli -->|"benchmark"| node_analysis
node_generators -->|"emit manifest"| node_schemas
node_generators -->|"resolve family"| node_generator_registry
node_generators -->|"family impl"| node_tabular
node_generators -->|"family impl"| node_script_repair
node_generators -->|"family impl"| node_pipeline
node_generators -->|"family impl"| node_retrieval_workspace
node_runtime -->|"load workspace"| node_environment_loader
node_runtime -->|"actions"| node_tool_executor
node_runtime -->|"policy loop"| node_policy_layer
node_policy_layer -->|"baseline"| node_agents
node_runtime -->|"trajectory"| node_schemas
node_evaluators -->|"resolve"| node_evaluator_registry
node_evaluators -->|"result"| node_schemas
node_evaluators -->|"family eval"| node_tabular
node_evaluators -->|"family eval"| node_script_repair
node_evaluators -->|"family eval"| node_pipeline
node_evaluators -->|"family eval"| node_retrieval_workspace
node_analysis -->|"consume"| node_schemas
node_analysis -->|"aggregate"| node_benchmarking
node_utils -.->|"helpers"| node_generators
node_utils -.->|"helpers"| node_runtime
node_utils -.->|"helpers"| node_analysis

click node_cli "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/cli.py"
click node_schemas "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/schemas/common.py"
click node_generators "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/base.py"
click node_generator_registry "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/registry.py"
click node_runtime "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/runtime/runner.py"
click node_environment_loader "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/runtime/environment.py"
click node_tool_executor "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/runtime/tools.py"
click node_policy_layer "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/runtime/policy.py"
click node_agents "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/agents/base.py"
click node_evaluators "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/evaluators/base.py"
click node_evaluator_registry "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/evaluators/registry.py"
click node_analysis "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/analysis/artifacts.py"
click node_benchmarking "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/analysis/benchmarking.py"
click node_tabular "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/tabular.py"
click node_script_repair "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/script_repair.py"
click node_pipeline "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/pipeline_completion.py"
click node_retrieval_workspace "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/retrieval_workspace.py"
click node_utils "https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/utils/scratch.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_schemas,node_generators,node_generator_registry,node_runtime,node_environment_loader,node_tool_executor,node_policy_layer,node_evaluators,node_evaluator_registry,node_utils toneBlue
class node_tabular,node_script_repair,node_pipeline,node_retrieval_workspace toneAmber
class node_cli,node_agents,node_analysis,node_benchmarking toneMint
```

## Components

### Core system

Contracts, generation, execution, and scoring for synthetic workspaces.

- [Schemas](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/schemas/common.py): contracts · Shared typed structures for specs, manifests, trajectories, and evaluator outputs. · `src/synthetic_workspace_gym/schemas/common.py`
- [Generators](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/base.py): build envs · Creates environment directories, manifests, visible files, and hidden evaluator assets. · `src/synthetic_workspace_gym/generators/base.py`
- [Gen registry](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/registry.py): resolver · Resolves generator implementations by family or scenario. · `src/synthetic_workspace_gym/generators/registry.py`
- [Runtime](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/runtime/runner.py): episode exec · Loads generated environments and runs tool-based episodes in a scratch workspace. · `src/synthetic_workspace_gym/runtime/runner.py`
- [Loader](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/runtime/environment.py): workspace load · Copies the visible workspace into an isolated scratch area and prepares episode state. · `src/synthetic_workspace_gym/runtime/environment.py`
- [Tools](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/runtime/tools.py): action layer · Executes file and shell actions with local guardrails. · `src/synthetic_workspace_gym/runtime/tools.py`
- [Policy](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/runtime/policy.py): agent bridge · Connects the runner to baseline policies and action selection. · `src/synthetic_workspace_gym/runtime/policy.py`
- [Evaluators](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/evaluators/base.py): trusted scoring · Resolves and runs hidden evaluators against the final workspace and assets. · `src/synthetic_workspace_gym/evaluators/base.py`
- [Eval registry](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/evaluators/registry.py): resolver · Dynamic evaluator lookup from manifest entrypoints and family registration. · `src/synthetic_workspace_gym/evaluators/registry.py`
- [Utils](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/utils/scratch.py): support · Shared I/O, path, and scratch-space helpers used across the repo. · `src/synthetic_workspace_gym/utils/scratch.py`

### Environment families

Task-family implementations for concrete synthetic environments.

- [Tabular](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/tabular.py): family · Tabular environment generation and evaluation family. · `src/synthetic_workspace_gym/generators/tabular.py`
- [Script repair](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/script_repair.py): family · Script-fixing environment family with its own scenarios and evaluator. · `src/synthetic_workspace_gym/generators/script_repair.py`
- [Pipeline](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/pipeline_completion.py): family · Pipeline completion environment family and scenarios. · `src/synthetic_workspace_gym/generators/pipeline_completion.py`
- [Retrieval workspace](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/generators/retrieval_workspace.py): family · Retrieval-oriented workspace family with hidden assets and evaluation. · `src/synthetic_workspace_gym/generators/retrieval_workspace.py`

### Operations

User-facing entrypoints and result processing.

- [CLI](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/cli.py): entrypoint · Command-line orchestration for generation, execution, evaluation, and benchmarking. · `src/synthetic_workspace_gym/cli.py`
- [Agents](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/agents/base.py): baselines · Minimal scripted and compatibility baselines used for episodes and benchmarks. · `src/synthetic_workspace_gym/agents/base.py`
- [Analysis](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/analysis/artifacts.py): artifacts · Exports episode artifacts and aggregates benchmark results. · `src/synthetic_workspace_gym/analysis/artifacts.py`
- [Benchmarking](https://github.com/ydnyshhh/synthetic-workspace-gym/blob/main/src/synthetic_workspace_gym/analysis/benchmarking.py): summaries · Normalizes run rows and produces grouped benchmark summaries. · `src/synthetic_workspace_gym/analysis/benchmarking.py`

## Connections

- CLI → Generators: generate · Builds environment instances from specs and scenario choices.
- CLI → Runtime: run · Executes an agent episode in a generated environment.
- CLI → Evaluators: evaluate · Runs trusted scoring on a workspace.
- CLI → Analysis: benchmark · Exports artifacts and sweeps baselines across environments.
- Generators → Schemas: emit manifest · Produces EnvironmentSpec and EnvironmentManifest data.
- Generators → Gen registry: resolve family · Selects family-specific generator implementations.
- Generators → Tabular: family impl · Generates tabular environments and scenarios.
- Generators → Script repair: family impl · Generates script repair environments and scenarios.
- Generators → Pipeline: family impl · Generates pipeline environments and scenarios.
- Generators → Retrieval workspace: family impl · Generates retrieval workspace environments and scenarios.
- Runtime → Loader: load workspace · Prepares scratch copies of visible files for an episode.
- Runtime → Tools: actions · Applies file edits and shell/python operations.
- Runtime → Policy: policy loop · Queries the selected baseline policy for next actions.
- Policy → Agents: baseline · Uses scripted and compatibility agent implementations.
- Runtime → Schemas: trajectory · Produces step-level records and episode outputs.
- Evaluators → Eval registry: resolve · Loads the manifest-specified evaluator entrypoint.
- Evaluators → Schemas: result · Produces EvaluatorResult and subscores.
- Evaluators → Tabular: family eval · Tabular scoring logic.
- Evaluators → Script repair: family eval · Script repair scoring logic.
- Evaluators → Pipeline: family eval · Pipeline scoring logic.
- Evaluators → Retrieval workspace: family eval · Retrieval workspace scoring logic.
- Analysis → Schemas: consume · Reads manifests, trajectories, diffs, and evaluator outputs.
- Analysis → Benchmarking: aggregate · Builds grouped summaries across runs.
- Utils → Generators: helpers · Supports path handling and workspace creation.
- Utils → Runtime: helpers · Supports scratch-space and file operations.
- Utils → Analysis: helpers · Supports artifact I/O and path management.
