How physical-intelligence/openpi works
openpi is a robotics vision-language-action model project: it supports π₀, π₀-FAST, and π₀.₅, with base checkpoints, fine-tuning, and robot-specific inference. The principal runtime path is a robot-side client sending observations over WebSocket to a policy server, which adapts inputs, runs a model, and returns actions. Training is a separate data-to-model workflow. Some subsystem links rely on README claims and tree paths because their source was not sampled; uncertain call wiring is omitted.
Main entry points: policies Robot policy adapters · data_loader.py Data loading · runtime.py Agent runtime · models JAX model family
Components 26
Training and data
- Training entrypoints
scripts/train.py - PyTorch training
scripts/train_pytorch.py - Data loading
src/openpi/training/data_loader.py - DROID RLDS data
src/openpi/training/droid_rlds_dataset.py - Data transforms
src/openpi/transforms.py - Normalization stats
scripts/compute_norm_stats.py - Optimization and checkpoints
src/openpi/training
VLA models
- JAX model family
src/openpi/models - PyTorch π₀ models
src/openpi/models_pytorch/pi0_pytorch.py - Action tokenizers
src/openpi/models/tokenizer.py - Checkpoint downloads
src/openpi/shared/download.py
Policy inference
- Policy serving entrypoint
scripts/serve_policy.py - Policy construction
src/openpi/policies/policy_config.py - Robot policy adapters
src/openpi/policies - ALOHA adapter
src/openpi/policies/aloha_policy.py - DROID adapter
src/openpi/policies/droid_policy.py - LIBERO adapter
src/openpi/policies/libero_policy.py - WebSocket policy server
src/openpi/serving/websocket_policy_server.py
Robot client runtime
- WebSocket client policy
packages/openpi-client/src/openpi_client/websocket_client_policy.py - Agent runtime
packages/openpi-client/src/openpi_client/runtime/runtime.py - Policy agent
packages/openpi-client/src/openpi_client/runtime/agents/policy_agent.py - Environment interface
packages/openpi-client/src/openpi_client/runtime/environment.py - Action subscriber
packages/openpi-client/src/openpi_client/runtime/subscriber.py
Other components
- Robot or application
- Robot datasets
- Checkpoint storage
Connections 32
- Robot datasets → Data loading: provides samples
- DROID RLDS data → Data loading: supplies DROID data
- Data loading → Data transforms: applies transforms
- Training entrypoints → Data loading: loads batches
- PyTorch training → Data loading: loads batches
- Training entrypoints → Optimization and checkpoints: trains and checkpoints
- PyTorch training → Optimization and checkpoints: trains and checkpoints
- Normalization stats → Robot datasets: computes statistics
- Optimization and checkpoints → Checkpoint storage: loads or saves checkpoints
- Training entrypoints → JAX model family: trains model
- PyTorch training → PyTorch π₀ models: trains model
- JAX model family → Action tokenizers: uses tokenizers
- PyTorch π₀ models → Action tokenizers: uses tokenizers
- Action tokenizers → Checkpoint downloads: fetches tokenizer assets
- Policy serving entrypoint → Policy construction: constructs policy
- Policy construction → Robot policy adapters: configures adapter
- Robot policy adapters → ALOHA adapter: selects platform adapter
- Robot policy adapters → DROID adapter: selects platform adapter
- Robot policy adapters → LIBERO adapter: selects platform adapter
- Policy serving entrypoint → WebSocket policy server: serves policy
- WebSocket policy server → Robot policy adapters: invokes inference
- Robot policy adapters → JAX model family: runs model
- Robot policy adapters → PyTorch π₀ models: runs model
- JAX model family → Checkpoint downloads: loads model assets
- Robot or application → Agent runtime: provides observations
- Agent runtime → Environment interface: reads environment
- Agent runtime → Policy agent: dispatches agent
- Policy agent → WebSocket client policy: requests inference
- WebSocket client policy → WebSocket policy server: sends observations
- WebSocket policy server → WebSocket client policy: returns actions
- Agent runtime → Action subscriber: dispatches actions
- Action subscriber → Robot or application: delivers actions