# zzongzheng0918/ta-vla architecture

> Architecture diagram of the GitHub repository zzongzheng0918/ta-vla, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zzongzheng0918/ta-vla
- Repository: https://github.com/zzongzheng0918/ta-vla

## Overview

TA-VLA extends openpi with torque-aware vision-language-action training and deployment. A typical path is robot/dataset observations (including effort) through configured transforms and normalization, into the Pi0 model for training or action inference; configured policies expose the model, and the WebSocket server and client support remote deployment. Training configuration also covers model variants and torque input/output modes; README documents historical-effort buffers in deployment, but their implementation is not in this tree. Edges reflect shown imports/calls or README claims; connections to unsampled subsystems are omitted where callers are not evidenced.

Main entry points: [`src/openpi/training/config.py`](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/config.py) (Training configs), [`src/openpi/models/pi0.py`](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/pi0.py) (Pi0 model), [`packages/openpi-client/src/openpi_client/websocket_client_policy.py`](https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/websocket_client_policy.py) (WebSocket policy client), [`src/openpi/policies/policy_config.py`](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/policies/policy_config.py) (Policy configuration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_dataset["LeRobot dataset<br/>[data_loader.py]"]
  node_transforms["Data transforms<br/>[transforms.py]"]
  node_normalization["Normalization stats<br/>[normalize.py]"]
end

subgraph group_model["Modeling"]
  node_effort["Torque modes<br/>[effort_type.py]"]
  node_observation["Observation schema<br/>[model.py]"]
  node_pi0["Pi0 model<br/>[pi0.py]"]
  node_gemma["Gemma backbone<br/>[gemma.py]"]
  node_siglip["SigLIP vision<br/>[siglip.py]"]
  node_fast["Pi0 FAST<br/>[pi0_fast.py]"]
  node_tokenizer["Tokenizers<br/>[tokenizer.py]"]
  node_lora["LoRA layers<br/>[lora.py]"]
end

subgraph group_train["Training"]
  node_trainconfig["Training configs<br/>[config.py]"]
  node_optimizer["Optimizer<br/>[optimizer.py]"]
  node_sharding["Device sharding<br/>[sharding.py]"]
  node_weights["Weight loading<br/>[weight_loaders.py]"]
  node_checkpoint[("Checkpoints<br/>[checkpoints.py]")]
end

subgraph group_deploy["Policy deployment"]
  node_policies["Robot policies<br/>[tavla_policy.py]"]
  node_policyconfig["Policy configuration<br/>[policy_config.py]"]
  node_policy["Inference policy<br/>[policy.py]"]
  node_server["WebSocket server"]
end

subgraph group_client["Client runtime"]
  node_clientpolicy["WebSocket policy client"]
  node_broker["Action chunk broker"]
  node_runtime["Agent runtime<br/>[runtime.py]"]
  node_agent["Policy agent<br/>[policy_agent.py]"]
  node_environment["Environment interface<br/>[environment.py]"]
  node_subscriber["Runtime subscriber<br/>[subscriber.py]"]
  node_serialization["Msgpack arrays<br/>[msgpack_numpy.py]"]
end

node_actor(("Robot client"))

node_dataset -->|"transforms"| node_transforms
node_trainconfig -->|"configures"| node_transforms
node_trainconfig -->|"loads stats"| node_normalization
node_transforms -->|"applies stats"| node_normalization
node_dataset -->|"batches as"| node_observation
node_trainconfig -->|"selects modes"| node_effort
node_observation -.->|"represents input"| node_effort
node_pi0 -->|"uses modes"| node_effort
node_pi0 -->|"builds backbone"| node_gemma
node_pi0 -->|"builds vision"| node_siglip
node_gemma -->|"uses layers"| node_lora
node_fast -.->|"uses tokens"| node_tokenizer
node_trainconfig -->|"configures model"| node_pi0
node_trainconfig -->|"configures model"| node_fast
node_weights -->|"loads weights"| node_pi0
node_policyconfig -->|"reads config"| node_trainconfig
node_policyconfig -->|"loads checkpoint"| node_checkpoint
node_policyconfig -->|"creates policy"| node_policy
node_policy -->|"runs inference"| node_pi0
node_policyconfig -->|"selects robot policy"| node_policies
node_server -->|"serves inference"| node_policy
node_server -->|"encodes messages"| node_serialization
node_actor -.->|"sends observations"| node_clientpolicy
node_clientpolicy -.->|"requests actions"| node_server
node_clientpolicy -->|"encodes messages"| node_serialization
node_runtime -->|"dispatches agent"| node_agent
node_runtime -->|"uses environment"| node_environment
node_runtime -->|"uses subscriber"| node_subscriber
node_agent -.->|"invokes policy"| node_clientpolicy
node_broker -->|"uses policy"| node_clientpolicy

click node_dataset "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/data_loader.py"
click node_transforms "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/transforms.py"
click node_normalization "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/shared/normalize.py"
click node_effort "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/shared/effort_type.py"
click node_observation "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/model.py"
click node_pi0 "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/pi0.py"
click node_gemma "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/gemma.py"
click node_siglip "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/siglip.py"
click node_fast "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/pi0_fast.py"
click node_tokenizer "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/tokenizer.py"
click node_lora "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/lora.py"
click node_trainconfig "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/config.py"
click node_optimizer "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/optimizer.py"
click node_sharding "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/sharding.py"
click node_weights "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/weight_loaders.py"
click node_checkpoint "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/checkpoints.py"
click node_policies "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/policies/tavla_policy.py"
click node_policyconfig "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/policies/policy_config.py"
click node_policy "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/policies/policy.py"
click node_server "https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/serving/websocket_policy_server.py"
click node_clientpolicy "https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/websocket_client_policy.py"
click node_broker "https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/action_chunk_broker.py"
click node_runtime "https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/runtime/runtime.py"
click node_agent "https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/runtime/agents/policy_agent.py"
click node_environment "https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/runtime/environment.py"
click node_subscriber "https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/runtime/subscriber.py"
click node_serialization "https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/msgpack_numpy.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset,node_transforms,node_normalization,node_actor toneBlue
class node_effort,node_observation,node_pi0,node_gemma,node_siglip,node_fast,node_tokenizer,node_lora toneAmber
class node_trainconfig,node_optimizer,node_sharding,node_weights,node_checkpoint toneMint
class node_policies,node_policyconfig,node_policy,node_server toneRose
class node_clientpolicy,node_broker,node_runtime,node_agent,node_environment,node_subscriber,node_serialization toneIndigo
```

## Components

### Data preparation

- [LeRobot dataset](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/data_loader.py): `src/openpi/training/data_loader.py`
- [Data transforms](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/transforms.py): `src/openpi/transforms.py`
- [Normalization stats](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/shared/normalize.py): `src/openpi/shared/normalize.py`

### Modeling

- [Torque modes](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/shared/effort_type.py): `src/openpi/shared/effort_type.py`
- [Observation schema](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/model.py): `src/openpi/models/model.py`
- [Pi0 model](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/pi0.py): `src/openpi/models/pi0.py`
- [Gemma backbone](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/gemma.py): `src/openpi/models/gemma.py`
- [SigLIP vision](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/siglip.py): `src/openpi/models/siglip.py`
- [Pi0 FAST](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/pi0_fast.py): `src/openpi/models/pi0_fast.py`
- [Tokenizers](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/tokenizer.py): `src/openpi/models/tokenizer.py`
- [LoRA layers](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/models/lora.py): `src/openpi/models/lora.py`

### Training

- [Training configs](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/config.py): `src/openpi/training/config.py`
- [Optimizer](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/optimizer.py): `src/openpi/training/optimizer.py`
- [Device sharding](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/sharding.py): `src/openpi/training/sharding.py`
- [Weight loading](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/weight_loaders.py): `src/openpi/training/weight_loaders.py`
- [Checkpoints](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/training/checkpoints.py): `src/openpi/training/checkpoints.py`

### Policy deployment

- [Robot policies](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/policies/tavla_policy.py): `src/openpi/policies/tavla_policy.py`
- [Policy configuration](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/policies/policy_config.py): `src/openpi/policies/policy_config.py`
- [Inference policy](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/policies/policy.py): `src/openpi/policies/policy.py`
- [WebSocket server](https://github.com/zzongzheng0918/ta-vla/blob/main/src/openpi/serving/websocket_policy_server.py): `src/openpi/serving/websocket_policy_server.py`

### Client runtime

- [WebSocket policy client](https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/websocket_client_policy.py): `packages/openpi-client/src/openpi_client/websocket_client_policy.py`
- [Action chunk broker](https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/action_chunk_broker.py): `packages/openpi-client/src/openpi_client/action_chunk_broker.py`
- [Agent runtime](https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/runtime/runtime.py): `packages/openpi-client/src/openpi_client/runtime/runtime.py`
- [Policy agent](https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/runtime/agents/policy_agent.py): `packages/openpi-client/src/openpi_client/runtime/agents/policy_agent.py`
- [Environment interface](https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/runtime/environment.py): `packages/openpi-client/src/openpi_client/runtime/environment.py`
- [Runtime subscriber](https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/runtime/subscriber.py): `packages/openpi-client/src/openpi_client/runtime/subscriber.py`
- [Msgpack arrays](https://github.com/zzongzheng0918/ta-vla/blob/main/packages/openpi-client/src/openpi_client/msgpack_numpy.py): `packages/openpi-client/src/openpi_client/msgpack_numpy.py`

### Other components

- **Robot client**

## Connections

- LeRobot dataset → Data transforms: transforms (evidence: `src/openpi/training/data_loader.py`)
- Training configs → Data transforms: configures (evidence: `src/openpi/training/config.py`)
- Training configs → Normalization stats: loads stats (evidence: `src/openpi/training/config.py`)
- Data transforms → Normalization stats: applies stats (evidence: `src/openpi/transforms.py`)
- LeRobot dataset → Observation schema: batches as (evidence: `src/openpi/training/data_loader.py`)
- Training configs → Torque modes: selects modes (evidence: `src/openpi/training/config.py`)
- Observation schema → Torque modes: represents input (evidence: `src/openpi/models/model.py`)
- Pi0 model → Torque modes: uses modes (evidence: `src/openpi/models/pi0.py`)
- Pi0 model → Gemma backbone: builds backbone (evidence: `src/openpi/models/pi0.py`)
- Pi0 model → SigLIP vision: builds vision (evidence: `src/openpi/models/pi0.py`)
- Gemma backbone → LoRA layers: uses layers (evidence: `src/openpi/models/gemma.py`)
- Pi0 FAST → Tokenizers: uses tokens
- Training configs → Pi0 model: configures model (evidence: `src/openpi/training/config.py`)
- Training configs → Pi0 FAST: configures model (evidence: `src/openpi/training/config.py`)
- Weight loading → Pi0 model: loads weights (evidence: `src/openpi/training/weight_loaders.py`)
- Policy configuration → Training configs: reads config (evidence: `src/openpi/policies/policy_config.py`)
- Policy configuration → Checkpoints: loads checkpoint (evidence: `src/openpi/policies/policy_config.py`)
- Policy configuration → Inference policy: creates policy (evidence: `src/openpi/policies/policy_config.py`)
- Inference policy → Pi0 model: runs inference (evidence: `src/openpi/policies/policy.py`)
- Policy configuration → Robot policies: selects robot policy (evidence: `src/openpi/policies/policy_config.py`)
- WebSocket server → Inference policy: serves inference (evidence: `src/openpi/serving/websocket_policy_server.py`)
- WebSocket server → Msgpack arrays: encodes messages (evidence: `src/openpi/serving/websocket_policy_server.py`)
- Robot client → WebSocket policy client: sends observations
- WebSocket policy client → WebSocket server: requests actions
- WebSocket policy client → Msgpack arrays: encodes messages (evidence: `packages/openpi-client/src/openpi_client/websocket_client_policy.py`)
- Agent runtime → Policy agent: dispatches agent (evidence: `packages/openpi-client/src/openpi_client/runtime/runtime.py`)
- Agent runtime → Environment interface: uses environment (evidence: `packages/openpi-client/src/openpi_client/runtime/runtime.py`)
- Agent runtime → Runtime subscriber: uses subscriber (evidence: `packages/openpi-client/src/openpi_client/runtime/runtime.py`)
- Policy agent → WebSocket policy client: invokes policy
- Action chunk broker → WebSocket policy client: uses policy (evidence: `packages/openpi-client/src/openpi_client/action_chunk_broker.py`)
