How robo-arena/roboarena works
1. What this repository is
- This is a small Python package that acts as an integration layer for RoboArena: it standardizes how a robot policy is exposed over the network so it can be queried by the RoboArena evaluator.
- It is not the benchmark itself; it provides the policy-serving client/server boundary and a minimal local test harness.
2. Primary architectural boundary
- The main boundary is remote policy serving: a policy implementation runs in one process, and an evaluator/client communicates with it over a WebSocket connection.
- This boundary matters because it separates model inference from the benchmarking runtime and allows distributed evaluation.
3. Core policy contract
- roboarena/policy.py defines the policy interface, centered on a BasePolicy abstraction.
- Any custom model intended for RoboArena must conform to this interface so the server can call it consistently regardless of the underlying ML stack.
4. Policy server subsystem
- roboarena/policy_server.py is the main serving layer.
- It exposes WebsocketPolicyServer and a PolicyServerConfig that describes what the policy needs and returns: image resolution, wrist camera usage, number of external cameras, stereo support, and action-space type.
- This config is the contract used to minimize data sent over the wire by only transmitting required observations.
5. Policy client subsystem
- roboarena/policy_client.py is the consumer side of the architecture.
- It queries the policy server and is used for local validation or as a stand-in for the evaluator-side request flow.
- Together, client and server define the request/response loop around policy inference.
6. Serialization and observation transport
- roboarena/utils/msgpack_numpy.py suggests the repo uses MessagePack with NumPy support for efficient transport of numeric tensors/arrays.
- This is important because robot observations and actions are typically array-heavy, so serialization format affects latency and bandwidth.
7. Validation and smoke testing
- scripts/test_policy_server.py is a standalone smoke test for the server path.
- Its role is to verify the server can be started and queried end-to-end before connecting to RoboArena or simulation-based evaluation.
8. Repository layout
- roboarena/ contains the importable package and the reusable runtime components.
- scripts/ contains operational checks rather than reusable library code.
- docs/ contains product imagery and likely supporting documentation, but it is not part of the execution path.
9. Runtime and tooling
- The project is Python-based and managed with uv (pyproject.toml, uv.lock, .python-version).
- The README shows both editable installation and direct module execution, which indicates the package is meant to be run locally as a lightweight service.
10. External ecosystem boundaries
- RoboArena is positioned as the upstream evaluation system, while DROID simulation evaluation and openpi are external downstream/upstream workflows.
- This repo’s architectural role is to bridge a trained policy into those environments through a standardized serving interface.
11. How data likely flows
- An evaluator or client sends a compact observation payload to the policy server.
- The server deserializes it, converts it to the BasePolicy input format, runs inference, serializes the action output, and returns it over the WebSocket connection.
12. What to draw in an architecture diagram
- Draw four main blocks: policy implementation, websocket policy server, client/evaluator, and serialization utilities.
- Show PolicyServerConfig as metadata/control information that shapes the observation/action contract.
- Show external arrows to RoboArena evaluation and simulation environments as the consumers of the serving interface.
Main entry points: policy_server.py WS server · policy_client.py Policy client · policy.py BasePolicy · msgpack_numpy.py Msgpack NumPy
Components 12
Core package
Reusable runtime components that define policy serving and the policy contract.
- BasePolicy · Core policy contract that every RoboArena policy must implement.
roboarena/policy.py - Policy impl · A concrete model or policy implementation that plugs into the BasePolicy contract.
- Server config · Describes observation needs and action-space shape for efficient wire transfer.
roboarena/policy_server.py - WS server · WebSocket serving layer that receives observations and returns actions.
roboarena/policy_server.py - Obs adapter · Converts compact network payloads into the input format expected by the policy.
roboarena/policy_server.py - Action encoder · Serializes policy outputs back onto the wire for the evaluator or client.
roboarena/policy_server.py - Policy client · Local consumer that exercises the same request/response flow as the evaluator.
roboarena/policy_client.py - Msgpack NumPy · MessagePack-based transport helper with NumPy support for array-heavy observations and actions.
roboarena/utils/msgpack_numpy.py
Tooling
Local validation and operational checks.
- Smoke test · Standalone end-to-end check that starts the server path and verifies it can be queried.
scripts/test_policy_server.py
External systems
Upstream or downstream systems that consume the serving interface.
- RoboArena · Upstream evaluation system that queries the served policy over the network.
- Sim eval · External simulation-based evaluation workflow that can reuse the same serving boundary.
- openpi · External policy-training or policy-runtime workflow that can feed implementations into RoboArena serving.
Connections 13
- Policy impl → BasePolicy: implements · A concrete policy conforms to the shared interface.
- Server config → WS server: configures · Server behavior and observation contract are shaped by the serving config.
- WS server → BasePolicy: invokes · The server calls the policy for inference.
- WS server → Obs adapter: decodes · Incoming observation payloads are transformed into policy inputs.
- Obs adapter → Msgpack NumPy: uses · Compact array transport is handled through MessagePack with NumPy support.
- WS server → Action encoder: encodes · Policy outputs are serialized back into network messages.
- Action encoder → Msgpack NumPy: uses · Responses use the same efficient numeric serialization layer.
- Policy client → WS server: queries · The client exercises the same request/response loop as the evaluator.
- Smoke test → WS server: verifies · Smoke test starts and queries the server end-to-end.
- RoboArena → WS server: consumes · RoboArena is the main upstream consumer of the serving interface.
- Sim eval → WS server: consumes · Simulation-based evaluation can also drive the same policy server.
- openpi → Policy impl: supplies · External policy workflows can provide implementations that satisfy the contract.
- WS server → Policy client: mirrors · Client and server define the same boundary from opposite sides.