How zzongzheng0918/ta-vla works
TA-VLA extends openpi with torque-aware vision-language-action training and deployment. A typical path is robot/dataset observations (including effort) through configured transforms and normalization, into the Pi0 model for training or action inference; configured policies expose the model, and the WebSocket server and client support remote deployment. Training configuration also covers model variants and torque input/output modes; README documents historical-effort buffers in deployment, but their implementation is not in this tree. Edges reflect shown imports/calls or README claims; connections to unsampled subsystems are omitted where callers are not evidenced.
Main entry points: config.py Training configs · pi0.py Pi0 model · websocket_client_policy.py WebSocket policy client · policy_config.py Policy configuration
Components 28
Data preparation
- LeRobot dataset
src/openpi/training/data_loader.py - Data transforms
src/openpi/transforms.py - Normalization stats
src/openpi/shared/normalize.py
Modeling
- Torque modes
src/openpi/shared/effort_type.py - Observation schema
src/openpi/models/model.py - Pi0 model
src/openpi/models/pi0.py - Gemma backbone
src/openpi/models/gemma.py - SigLIP vision
src/openpi/models/siglip.py - Pi0 FAST
src/openpi/models/pi0_fast.py - Tokenizers
src/openpi/models/tokenizer.py - LoRA layers
src/openpi/models/lora.py
Training
- Training configs
src/openpi/training/config.py - Optimizer
src/openpi/training/optimizer.py - Device sharding
src/openpi/training/sharding.py - Weight loading
src/openpi/training/weight_loaders.py - Checkpoints
src/openpi/training/checkpoints.py
Policy deployment
- Robot policies
src/openpi/policies/tavla_policy.py - Policy configuration
src/openpi/policies/policy_config.py - Inference policy
src/openpi/policies/policy.py - WebSocket server
src/openpi/serving/websocket_policy_server.py
Client runtime
- WebSocket policy client
packages/openpi-client/src/openpi_client/websocket_client_policy.py - Action chunk broker
packages/openpi-client/src/openpi_client/action_chunk_broker.py - Agent runtime
packages/openpi-client/src/openpi_client/runtime/runtime.py - Policy agent
packages/openpi-client/src/openpi_client/runtime/agents/policy_agent.py - Environment interface
packages/openpi-client/src/openpi_client/runtime/environment.py - Runtime subscriber
packages/openpi-client/src/openpi_client/runtime/subscriber.py - Msgpack arrays
packages/openpi-client/src/openpi_client/msgpack_numpy.py
Other components
- Robot client
Connections 30
- LeRobot dataset → Data transforms: transforms
src/openpi/training/data_loader.py - Training configs → Data transforms: configures
src/openpi/training/config.py - Training configs → Normalization stats: loads stats
src/openpi/training/config.py - Data transforms → Normalization stats: applies stats
src/openpi/transforms.py - LeRobot dataset → Observation schema: batches as
src/openpi/training/data_loader.py - Training configs → Torque modes: selects modes
src/openpi/training/config.py - Observation schema → Torque modes: represents input
src/openpi/models/model.py - Pi0 model → Torque modes: uses modes
src/openpi/models/pi0.py - Pi0 model → Gemma backbone: builds backbone
src/openpi/models/pi0.py - Pi0 model → SigLIP vision: builds vision
src/openpi/models/pi0.py - Gemma backbone → LoRA layers: uses layers
src/openpi/models/gemma.py - Pi0 FAST → Tokenizers: uses tokens
- Training configs → Pi0 model: configures model
src/openpi/training/config.py - Training configs → Pi0 FAST: configures model
src/openpi/training/config.py - Weight loading → Pi0 model: loads weights
src/openpi/training/weight_loaders.py - Policy configuration → Training configs: reads config
src/openpi/policies/policy_config.py - Policy configuration → Checkpoints: loads checkpoint
src/openpi/policies/policy_config.py - Policy configuration → Inference policy: creates policy
src/openpi/policies/policy_config.py - Inference policy → Pi0 model: runs inference
src/openpi/policies/policy.py - Policy configuration → Robot policies: selects robot policy
src/openpi/policies/policy_config.py - WebSocket server → Inference policy: serves inference
src/openpi/serving/websocket_policy_server.py - WebSocket server → Msgpack arrays: encodes messages
src/openpi/serving/websocket_policy_server.py - Robot client → WebSocket policy client: sends observations
- WebSocket policy client → WebSocket server: requests actions
- WebSocket policy client → Msgpack arrays: encodes messages
packages/openpi-client/src/openpi_client/websocket_client_policy.py - Agent runtime → Policy agent: dispatches agent
packages/openpi-client/src/openpi_client/runtime/runtime.py - Agent runtime → Environment interface: uses environment
packages/openpi-client/src/openpi_client/runtime/runtime.py - Agent runtime → Runtime subscriber: uses subscriber
packages/openpi-client/src/openpi_client/runtime/runtime.py - Policy agent → WebSocket policy client: invokes policy
- Action chunk broker → WebSocket policy client: uses policy
packages/openpi-client/src/openpi_client/action_chunk_broker.py