# rdi-berkeley/agents-last-exam architecture

> Architecture diagram of the GitHub repository rdi-berkeley/agents-last-exam, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rdi-berkeley/agents-last-exam
- Repository: https://github.com/rdi-berkeley/agents-last-exam

## Overview

Agents’ Last Exam (ALE) is an evaluation framework for running professional tasks with AI agents in provisioned sandboxes and scoring their resulting artifacts. The documented workflow pairs an agent, environment, and task; provisions the sandbox, stages inputs, runs the agent, stages hidden references only afterward, then grades and records the run. The graph reflects README-described capabilities and the supplied tree; orchestration and provider wiring are documented but their source was not sampled, so uncertain implementation-level links are omitted.

Main entry points: [`ale_run/orchestration/lifecycle.py`](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/lifecycle.py) (Run lifecycle), [`ale_run/environments/env.py`](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/environments/env.py) (Sandbox environment), [`ale_run/orchestration/runner.py`](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/runner.py) (Experiment runner), [`ale_run/agents`](https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/agents) (Agent harnesses)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Run orchestration"]
  node_cli["CLI<br/>[cli.py]"]
  node_runner["Experiment runner<br/>[runner.py]"]
  node_spec["Experiment specification<br/>[experiment_spec.py]"]
  node_lifecycle["Run lifecycle<br/>[lifecycle.py]"]
end

subgraph group_environments["Sandbox runtime"]
  node_environment["Sandbox environment<br/>[env.py]"]
  node_providers["Sandbox providers"]
  node_images["Sandbox images"]
  node_taskdata["Task data staging"]
  node_outputpull["Artifact collection<br/>[output_pull.py]"]
  node_executor["Execution backends"]
end

subgraph group_agents["Agent harnesses"]
  node_agentdeploy["Agent deployers"]
  node_agentharness["Agent harnesses"]
  node_cuabridge["CUA MCP bridge<br/>[index.js]"]
  node_vmmcp["VM MCP bridge<br/>[index.js]"]
end

subgraph group_evaluation["Tasks and grading"]
  node_taskloader["Task loader<br/>[loader.py]"]
  node_taskdriver["Task driver<br/>[driver.py]"]
  node_taskcatalog["Task catalog"]
  node_grader["Task evaluation"]
end

subgraph group_records["Run records"]
  node_writer["Run writer<br/>[run_writer.py]"]
  node_trajectory["Unified trajectory<br/>[trajectory.py]"]
  node_runresults[("Run outputs<br/>[run_writer.py]")]
end

node_operator(("Experiment operator"))

node_operator -->|"starts run"| node_cli
node_cli -->|"invokes"| node_runner
node_runner -->|"loads config"| node_spec
node_runner -->|"runs lifecycle"| node_lifecycle
node_lifecycle -->|"provisions"| node_environment
node_environment -->|"uses provider"| node_providers
node_environment -->|"boots image"| node_images
node_lifecycle -->|"stages inputs"| node_taskdata
node_lifecycle -->|"loads task"| node_taskloader
node_taskloader -->|"resolves task"| node_taskcatalog
node_lifecycle -->|"deploys agent"| node_agentdeploy
node_agentdeploy -->|"launches"| node_agentharness
node_lifecycle -->|"runs workload"| node_executor
node_agentharness -->|"uses desktop tools"| node_cuabridge
node_agentharness -.->|"uses remote tools"| node_vmmcp
node_cuabridge -->|"controls desktop"| node_environment
node_vmmcp -.->|"accesses sandbox"| node_environment
node_lifecycle -->|"stages reference"| node_taskdata
node_lifecycle -->|"invokes grading"| node_taskdriver
node_taskdriver -->|"evaluates outputs"| node_grader
node_environment -->|"provides artifacts"| node_outputpull
node_lifecycle -->|"records steps"| node_trajectory
node_lifecycle -->|"writes run data"| node_writer
node_writer -->|"stores results"| node_runresults

click node_cli "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/cli.py"
click node_runner "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/runner.py"
click node_spec "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/experiment_spec.py"
click node_lifecycle "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/lifecycle.py"
click node_writer "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/run_writer.py"
click node_environment "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/environments/env.py"
click node_providers "https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/environments/providers"
click node_images "https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/environments/images"
click node_taskdata "https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/environments/task_data"
click node_outputpull "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/environments/output_pull.py"
click node_executor "https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/executors"
click node_agentdeploy "https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/agents"
click node_agentharness "https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/agents"
click node_cuabridge "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/agents/_assets/cua_mcp_server/src/index.js"
click node_vmmcp "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/agents/_assets/vm_mcp_server/src/index.js"
click node_taskloader "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/tasks/loader.py"
click node_taskdriver "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/tasks/driver.py"
click node_taskcatalog "https://github.com/rdi-berkeley/agents-last-exam/tree/main/tasks"
click node_grader "https://github.com/rdi-berkeley/agents-last-exam/tree/main/tasks"
click node_trajectory "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/base_interface/trajectory.py"
click node_runresults "https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/run_writer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_runner,node_spec,node_lifecycle toneBlue
class node_environment,node_providers,node_images,node_taskdata,node_outputpull,node_executor toneAmber
class node_agentdeploy,node_agentharness,node_cuabridge,node_vmmcp toneMint
class node_taskloader,node_taskdriver,node_taskcatalog,node_grader toneRose
class node_writer,node_trajectory,node_runresults,node_operator toneIndigo
```

## Components

### Run orchestration

- [CLI](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/cli.py): `ale_run/cli.py`
- [Experiment runner](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/runner.py): `ale_run/orchestration/runner.py`
- [Experiment specification](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/experiment_spec.py): `ale_run/orchestration/experiment_spec.py`
- [Run lifecycle](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/lifecycle.py): `ale_run/orchestration/lifecycle.py`

### Sandbox runtime

- [Sandbox environment](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/environments/env.py): `ale_run/environments/env.py`
- [Sandbox providers](https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/environments/providers): `ale_run/environments/providers`
- [Sandbox images](https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/environments/images): `ale_run/environments/images`
- [Task data staging](https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/environments/task_data): `ale_run/environments/task_data`
- [Artifact collection](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/environments/output_pull.py): `ale_run/environments/output_pull.py`
- [Execution backends](https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/executors): `ale_run/executors`

### Agent harnesses

- [Agent deployers](https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/agents): `ale_run/agents`
- [Agent harnesses](https://github.com/rdi-berkeley/agents-last-exam/tree/main/ale_run/agents): `ale_run/agents`
- [CUA MCP bridge](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/agents/_assets/cua_mcp_server/src/index.js): `ale_run/agents/_assets/cua_mcp_server/src/index.js`
- [VM MCP bridge](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/agents/_assets/vm_mcp_server/src/index.js): `ale_run/agents/_assets/vm_mcp_server/src/index.js`

### Tasks and grading

- [Task loader](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/tasks/loader.py): `ale_run/tasks/loader.py`
- [Task driver](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/tasks/driver.py): `ale_run/tasks/driver.py`
- [Task catalog](https://github.com/rdi-berkeley/agents-last-exam/tree/main/tasks): `tasks`
- [Task evaluation](https://github.com/rdi-berkeley/agents-last-exam/tree/main/tasks): `tasks`

### Run records

- [Run writer](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/run_writer.py): `ale_run/orchestration/run_writer.py`
- [Unified trajectory](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/base_interface/trajectory.py): `ale_run/base_interface/trajectory.py`
- [Run outputs](https://github.com/rdi-berkeley/agents-last-exam/blob/main/ale_run/orchestration/run_writer.py): `ale_run/orchestration/run_writer.py`

### Other components

- **Experiment operator**

## Connections

- Experiment operator → CLI: starts run
- CLI → Experiment runner: invokes
- Experiment runner → Experiment specification: loads config
- Experiment runner → Run lifecycle: runs lifecycle
- Run lifecycle → Sandbox environment: provisions
- Sandbox environment → Sandbox providers: uses provider
- Sandbox environment → Sandbox images: boots image
- Run lifecycle → Task data staging: stages inputs
- Run lifecycle → Task loader: loads task
- Task loader → Task catalog: resolves task
- Run lifecycle → Agent deployers: deploys agent
- Agent deployers → Agent harnesses: launches
- Run lifecycle → Execution backends: runs workload
- Agent harnesses → CUA MCP bridge: uses desktop tools
- Agent harnesses → VM MCP bridge: uses remote tools
- CUA MCP bridge → Sandbox environment: controls desktop
- VM MCP bridge → Sandbox environment: accesses sandbox
- Run lifecycle → Task data staging: stages reference
- Run lifecycle → Task driver: invokes grading
- Task driver → Task evaluation: evaluates outputs
- Sandbox environment → Artifact collection: provides artifacts
- Run lifecycle → Unified trajectory: records steps
- Run lifecycle → Run writer: writes run data
- Run writer → Run outputs: stores results