# allenai/vla-evaluation-harness architecture

> Architecture diagram of the GitHub repository allenai/vla-evaluation-harness, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/allenai/vla-evaluation-harness
- Repository: https://github.com/allenai/vla-evaluation-harness

## Overview

vla-evaluation-harness evaluates vision-language-action models against robot-simulation benchmarks, with CLI and Python API entry points. The principal workflow configures a run, coordinates benchmark episodes and model-server requests, then collects and exports episode and aggregate results. The README also describes broad benchmark/model integrations, parallel evaluation, tracking, recording, and a separate leaderboard pipeline. The source excerpts are partial, so uncertain internal wiring is omitted; benchmark configurations themselves are not present in the supplied tree.

Main entry points: [`src/vla_eval/orchestrator.py`](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/orchestrator.py) (Run orchestrator), [`src/vla_eval/model_servers/base.py`](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/model_servers/base.py) (Model adapters), [`src/vla_eval/connection.py`](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/connection.py) (WebSocket connection), [`src/vla_eval/cli/main.py`](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/cli/main.py) (CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry points"]
  node_cli["CLI<br/>[main.py]"]
  node_api["Python API<br/>[api.py]"]
end

subgraph group_evaluation["Evaluation runtime"]
  node_orchestrator["Run orchestrator<br/>[orchestrator.py]"]
  node_registry["Benchmark registry<br/>[registry.py]"]
  node_runner["Episode runners<br/>[sync_runner.py]"]
  node_live_runner["Live runner<br/>[live_runner.py]"]
end

subgraph group_inference["Model inference"]
  node_connection["WebSocket connection<br/>[connection.py]"]
  node_server["Server runtime<br/>[serve.py]"]
  node_model["Model adapters<br/>[base.py]"]
  node_openvla["OpenVLA adapter<br/>[openvla.py]"]
  node_pi0["π₀ adapter<br/>[pi0.py]"]
  node_xvla["X-VLA adapter<br/>[xvla.py]"]
  node_protocol["Message codecs<br/>[messages.py]"]
end

subgraph group_results["Results and reporting"]
  node_collector["Result collector<br/>[collector.py]"]
  node_recording[("Episode recording<br/>[recording.py]")]
  node_export["Result export<br/>[export.py]"]
  node_tracking["Experiment tracking<br/>[tracking.py]"]
end

subgraph group_leaderboard["Leaderboard"]
  node_leaderboard_scripts["Leaderboard pipeline<br/>[scan.py]"]
  node_leaderboard_site["Leaderboard site<br/>[app.js]"]
end

node_benchmark["Simulation benchmark"]
node_user(("Evaluator"))
node_model_process(("Model process"))

node_user -->|"invokes"| node_cli
node_user -->|"calls"| node_api
node_cli -->|"starts run"| node_orchestrator
node_api -->|"starts run"| node_orchestrator
node_orchestrator -->|"resolves benchmark"| node_registry
node_orchestrator -->|"opens session"| node_connection
node_orchestrator -->|"runs episodes"| node_runner
node_orchestrator -->|"runs live episodes"| node_live_runner
node_runner -->|"steps environment"| node_benchmark
node_live_runner -->|"steps environment"| node_benchmark
node_runner -->|"sends observations"| node_connection
node_live_runner -->|"sends observations"| node_connection
node_connection -->|"WebSocket requests"| node_model_process
node_server -->|"dispatches inference"| node_model
node_model -->|"serves actions"| node_model_process
node_model -.->|"adapter family"| node_openvla
node_model -.->|"adapter family"| node_pi0
node_model -.->|"adapter family"| node_xvla
node_connection -->|"encodes messages"| node_protocol
node_orchestrator -->|"records episode results"| node_collector
node_orchestrator -->|"records run data"| node_recording
node_cli -->|"exports results"| node_export
node_export -->|"reads recordings"| node_recording
node_orchestrator -->|"emits run events"| node_tracking
node_export -->|"reports aggregates"| node_tracking
node_leaderboard_scripts -.->|"feeds leaderboard"| node_leaderboard_site

click node_cli "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/cli/main.py"
click node_api "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/api.py"
click node_orchestrator "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/orchestrator.py"
click node_registry "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/registry.py"
click node_runner "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/runners/sync_runner.py"
click node_live_runner "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/runners/live_runner.py"
click node_connection "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/connection.py"
click node_server "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/model_servers/serve.py"
click node_model "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/model_servers/base.py"
click node_openvla "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/model_servers/openvla.py"
click node_pi0 "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/model_servers/pi0.py"
click node_xvla "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/model_servers/xvla.py"
click node_protocol "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/protocol/messages.py"
click node_collector "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/results/collector.py"
click node_recording "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/recording.py"
click node_export "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/results/export.py"
click node_tracking "https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/tracking.py"
click node_leaderboard_scripts "https://github.com/allenai/vla-evaluation-harness/blob/main/leaderboard/scripts/scan.py"
click node_leaderboard_site "https://github.com/allenai/vla-evaluation-harness/blob/main/leaderboard/site/app.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_api toneBlue
class node_orchestrator,node_registry,node_runner,node_live_runner toneAmber
class node_connection,node_server,node_model,node_openvla,node_pi0,node_xvla,node_protocol toneMint
class node_collector,node_recording,node_export,node_tracking toneRose
class node_leaderboard_scripts,node_leaderboard_site,node_benchmark,node_user,node_model_process toneIndigo
```

## Components

### Entry points

- [CLI](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/cli/main.py): `src/vla_eval/cli/main.py`
- [Python API](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/api.py): `src/vla_eval/api.py`

### Evaluation runtime

- [Run orchestrator](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/orchestrator.py): `src/vla_eval/orchestrator.py`
- [Benchmark registry](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/registry.py): `src/vla_eval/registry.py`
- [Episode runners](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/runners/sync_runner.py): `src/vla_eval/runners/sync_runner.py`
- [Live runner](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/runners/live_runner.py): `src/vla_eval/runners/live_runner.py`

### Model inference

- [WebSocket connection](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/connection.py): `src/vla_eval/connection.py`
- [Server runtime](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/model_servers/serve.py): `src/vla_eval/model_servers/serve.py`
- [Model adapters](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/model_servers/base.py): `src/vla_eval/model_servers/base.py`
- [OpenVLA adapter](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/model_servers/openvla.py): `src/vla_eval/model_servers/openvla.py`
- [π₀ adapter](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/model_servers/pi0.py): `src/vla_eval/model_servers/pi0.py`
- [X-VLA adapter](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/model_servers/xvla.py): `src/vla_eval/model_servers/xvla.py`
- [Message codecs](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/protocol/messages.py): `src/vla_eval/protocol/messages.py`

### Results and reporting

- [Result collector](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/results/collector.py): `src/vla_eval/results/collector.py`
- [Episode recording](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/recording.py): `src/vla_eval/recording.py`
- [Result export](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/results/export.py): `src/vla_eval/results/export.py`
- [Experiment tracking](https://github.com/allenai/vla-evaluation-harness/blob/main/src/vla_eval/tracking.py): `src/vla_eval/tracking.py`

### Leaderboard

- [Leaderboard pipeline](https://github.com/allenai/vla-evaluation-harness/blob/main/leaderboard/scripts/scan.py): `leaderboard/scripts/scan.py`
- [Leaderboard site](https://github.com/allenai/vla-evaluation-harness/blob/main/leaderboard/site/app.js): `leaderboard/site/app.js`

### Other components

- **Simulation benchmark**
- **Evaluator**
- **Model process**

## Connections

- Evaluator → CLI: invokes
- Evaluator → Python API: calls
- CLI → Run orchestrator: starts run
- Python API → Run orchestrator: starts run
- Run orchestrator → Benchmark registry: resolves benchmark
- Run orchestrator → WebSocket connection: opens session
- Run orchestrator → Episode runners: runs episodes
- Run orchestrator → Live runner: runs live episodes
- Episode runners → Simulation benchmark: steps environment
- Live runner → Simulation benchmark: steps environment
- Episode runners → WebSocket connection: sends observations
- Live runner → WebSocket connection: sends observations
- WebSocket connection → Model process: WebSocket requests
- Server runtime → Model adapters: dispatches inference
- Model adapters → Model process: serves actions
- Model adapters → OpenVLA adapter: adapter family
- Model adapters → π₀ adapter: adapter family
- Model adapters → X-VLA adapter: adapter family
- WebSocket connection → Message codecs: encodes messages
- Run orchestrator → Result collector: records episode results
- Run orchestrator → Episode recording: records run data
- CLI → Result export: exports results
- Result export → Episode recording: reads recordings
- Run orchestrator → Experiment tracking: emits run events
- Result export → Experiment tracking: reports aggregates
- Leaderboard pipeline → Leaderboard site: feeds leaderboard