How allenai/vla-evaluation-harness works
vla-evaluation-harness evaluates vision-language-action models against robot-simulation benchmarks, with CLI and Python API entry points. The principal workflow configures a run, coordinates benchmark episodes and model-server requests, then collects and exports episode and aggregate results. The README also describes broad benchmark/model integrations, parallel evaluation, tracking, recording, and a separate leaderboard pipeline. The source excerpts are partial, so uncertain internal wiring is omitted; benchmark configurations themselves are not present in the supplied tree.
Main entry points: orchestrator.py Run orchestrator · base.py Model adapters · connection.py WebSocket connection · main.py CLI
Components 22
Entry points
- CLI
src/vla_eval/cli/main.py - Python API
src/vla_eval/api.py
Evaluation runtime
- Run orchestrator
src/vla_eval/orchestrator.py - Benchmark registry
src/vla_eval/registry.py - Episode runners
src/vla_eval/runners/sync_runner.py - Live runner
src/vla_eval/runners/live_runner.py
Model inference
- WebSocket connection
src/vla_eval/connection.py - Server runtime
src/vla_eval/model_servers/serve.py - Model adapters
src/vla_eval/model_servers/base.py - OpenVLA adapter
src/vla_eval/model_servers/openvla.py - π₀ adapter
src/vla_eval/model_servers/pi0.py - X-VLA adapter
src/vla_eval/model_servers/xvla.py - Message codecs
src/vla_eval/protocol/messages.py
Results and reporting
- Result collector
src/vla_eval/results/collector.py - Episode recording
src/vla_eval/recording.py - Result export
src/vla_eval/results/export.py - Experiment tracking
src/vla_eval/tracking.py
Leaderboard
- Leaderboard pipeline
leaderboard/scripts/scan.py - Leaderboard site
leaderboard/site/app.js
Other components
- Simulation benchmark
- Evaluator
- Model process
Connections 26
- Evaluator → CLI: invokes
- Evaluator → Python API: calls
- CLI → Run orchestrator: starts run
- Python API → Run orchestrator: starts run
- Run orchestrator → Benchmark registry: resolves benchmark
- Run orchestrator → WebSocket connection: opens session
- Run orchestrator → Episode runners: runs episodes
- Run orchestrator → Live runner: runs live episodes
- Episode runners → Simulation benchmark: steps environment
- Live runner → Simulation benchmark: steps environment
- Episode runners → WebSocket connection: sends observations
- Live runner → WebSocket connection: sends observations
- WebSocket connection → Model process: WebSocket requests
- Server runtime → Model adapters: dispatches inference
- Model adapters → Model process: serves actions
- Model adapters → OpenVLA adapter: adapter family
- Model adapters → π₀ adapter: adapter family
- Model adapters → X-VLA adapter: adapter family
- WebSocket connection → Message codecs: encodes messages
- Run orchestrator → Result collector: records episode results
- Run orchestrator → Episode recording: records run data
- CLI → Result export: exports results
- Result export → Episode recording: reads recordings
- Run orchestrator → Experiment tracking: emits run events
- Result export → Experiment tracking: reports aggregates
- Leaderboard pipeline → Leaderboard site: feeds leaderboard