How rdi-berkeley/agents-last-exam works
Agents’ Last Exam (ALE) is an evaluation framework for running professional tasks with AI agents in provisioned sandboxes and scoring their resulting artifacts. The documented workflow pairs an agent, environment, and task; provisions the sandbox, stages inputs, runs the agent, stages hidden references only afterward, then grades and records the run. The graph reflects README-described capabilities and the supplied tree; orchestration and provider wiring are documented but their source was not sampled, so uncertain implementation-level links are omitted.
Main entry points: lifecycle.py Run lifecycle · env.py Sandbox environment · runner.py Experiment runner · agents Agent harnesses
Components 22
Run orchestration
- CLI
ale_run/cli.py - Experiment runner
ale_run/orchestration/runner.py - Experiment specification
ale_run/orchestration/experiment_spec.py - Run lifecycle
ale_run/orchestration/lifecycle.py
Sandbox runtime
- Sandbox environment
ale_run/environments/env.py - Sandbox providers
ale_run/environments/providers - Sandbox images
ale_run/environments/images - Task data staging
ale_run/environments/task_data - Artifact collection
ale_run/environments/output_pull.py - Execution backends
ale_run/executors
Agent harnesses
- Agent deployers
ale_run/agents - Agent harnesses
ale_run/agents - CUA MCP bridge
ale_run/agents/_assets/cua_mcp_server/src/index.js - VM MCP bridge
ale_run/agents/_assets/vm_mcp_server/src/index.js
Tasks and grading
- Task loader
ale_run/tasks/loader.py - Task driver
ale_run/tasks/driver.py - Task catalog
tasks - Task evaluation
tasks
Run records
- Run writer
ale_run/orchestration/run_writer.py - Unified trajectory
ale_run/base_interface/trajectory.py - Run outputs
ale_run/orchestration/run_writer.py
Other components
- Experiment operator
Connections 24
- Experiment operator → CLI: starts run
- CLI → Experiment runner: invokes
- Experiment runner → Experiment specification: loads config
- Experiment runner → Run lifecycle: runs lifecycle
- Run lifecycle → Sandbox environment: provisions
- Sandbox environment → Sandbox providers: uses provider
- Sandbox environment → Sandbox images: boots image
- Run lifecycle → Task data staging: stages inputs
- Run lifecycle → Task loader: loads task
- Task loader → Task catalog: resolves task
- Run lifecycle → Agent deployers: deploys agent
- Agent deployers → Agent harnesses: launches
- Run lifecycle → Execution backends: runs workload
- Agent harnesses → CUA MCP bridge: uses desktop tools
- Agent harnesses → VM MCP bridge: uses remote tools
- CUA MCP bridge → Sandbox environment: controls desktop
- VM MCP bridge → Sandbox environment: accesses sandbox
- Run lifecycle → Task data staging: stages reference
- Run lifecycle → Task driver: invokes grading
- Task driver → Task evaluation: evaluates outputs
- Sandbox environment → Artifact collection: provides artifacts
- Run lifecycle → Unified trajectory: records steps
- Run lifecycle → Run writer: writes run data
- Run writer → Run outputs: stores results