# robbyant/lingbot-vla-v2 architecture

> Architecture diagram of the GitHub repository robbyant/lingbot-vla-v2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/robbyant/lingbot-vla-v2
- Repository: https://github.com/robbyant/lingbot-vla-v2

## Overview

LingBot-VLA 2.0 is a vision-language-action foundation model and robot policy stack. The main path is robot observations sent through a websocket client to a policy server, where the model produces actions for the robot; the server-side handoff is not verified in the supplied excerpts. The repository also supports heterogeneous robot/video training, multimodal data preparation, predictive depth and video supervision, and distributed post-training. Training and model links are grounded in the source index; unsampled runtime links are omitted or marked unverified.

Main entry points: [`tasks/vla/train_lingbotvla.py`](https://github.com/robbyant/lingbot-vla-v2/blob/main/tasks/vla/train_lingbotvla.py) (VLA trainer), [`deploy/lingbot_vla_v2_policy.py`](https://github.com/robbyant/lingbot-vla-v2/blob/main/deploy/lingbot_vla_v2_policy.py) (VLA policy), [`deploy/websocket_client_policy.py`](https://github.com/robbyant/lingbot-vla-v2/blob/main/deploy/websocket_client_policy.py) (Websocket client), [`lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla_v2.py`](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla_v2.py) (LingBot VLA)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Robot inference"]
  node_wsclient["Websocket client"]
  node_wsserver["Policy server"]
  node_policy["VLA policy"]
  node_eval["RoboTwin evaluator"]
end

subgraph group_model["VLA modeling"]
  node_model["LingBot VLA"]
  node_modelcore["VLA model core"]
  node_actionexpert["Action expert"]
  node_config["Model config"]
  node_vision["Vision teachers<br/>[module_utils.py]"]
end

subgraph group_data["Data pipeline"]
  node_chat["Chat templates<br/>[chat_template.py]"]
  node_multimodal["Multimodal data"]
  node_robotdata["Robot datasets"]
  node_transform["Data transforms<br/>[transform.py]"]
  node_normstats["Norm statistics"]
end

subgraph group_training["Training runtime"]
  node_trainer["VLA trainer"]
  node_parallel["Distributed parallelism"]
  node_parallelstate["Parallel state<br/>[parallel_state.py]"]
  node_offload["Activation offload<br/>[offloading.py]"]
  node_optimizer["Distributed Muon<br/>[dist_muon.py]"]
  node_checkpoint["Checkpointing<br/>[checkpointer.py]"]
end

node_robot(("Robot client"))

node_robot -.->|"sends observations"| node_wsclient
node_wsclient -.->|"requests inference"| node_wsserver
node_wsserver -.->|"dispatches policy"| node_policy
node_policy -->|"loads model"| node_model
node_policy -->|"uses config"| node_config
node_policy -->|"uses data utilities"| node_robotdata
node_trainer -->|"builds model"| node_model
node_trainer -->|"loads data"| node_robotdata
node_trainer -->|"parallelizes model"| node_parallel
node_trainer -->|"initializes state"| node_parallelstate
node_trainer -->|"configures offload"| node_offload
node_trainer -->|"uses optimizer"| node_optimizer
node_trainer -->|"saves checkpoints"| node_checkpoint
node_trainer -->|"uses vision teachers"| node_vision
node_trainer -.->|"computes normalization"| node_normstats
node_model -->|"uses config"| node_config
node_modelcore -->|"uses expert"| node_actionexpert
node_modelcore -->|"uses depth head"| node_vision
node_chat -->|"extends templates"| node_multimodal
node_eval -.->|"evaluates policy"| node_wsclient
node_robotdata -.->|"applies transforms"| node_transform

click node_wsclient "https://github.com/robbyant/lingbot-vla-v2/blob/main/deploy/websocket_client_policy.py"
click node_wsserver "https://github.com/robbyant/lingbot-vla-v2/blob/main/deploy/websocket_policy_server.py"
click node_policy "https://github.com/robbyant/lingbot-vla-v2/blob/main/deploy/lingbot_vla_v2_policy.py"
click node_model "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla_v2.py"
click node_modelcore "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla.py"
click node_actionexpert "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/models/vla/lingbot_vla/qwen2_action_expert.py"
click node_config "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/models/vla/lingbot_vla/configuration_lingbot_vla.py"
click node_vision "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/models/vla/vision_models/module_utils.py"
click node_chat "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/data/chat_template.py"
click node_multimodal "https://github.com/robbyant/lingbot-vla-v2/tree/main/lingbotvla/data/multimodal"
click node_robotdata "https://github.com/robbyant/lingbot-vla-v2/tree/main/lingbotvla/data/vla_data"
click node_transform "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/data/vla_data/transform.py"
click node_trainer "https://github.com/robbyant/lingbot-vla-v2/blob/main/tasks/vla/train_lingbotvla.py"
click node_parallel "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/distributed/torch_parallelize.py"
click node_parallelstate "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/distributed/parallel_state.py"
click node_offload "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/distributed/offloading.py"
click node_optimizer "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/optim/flex_shard/dist_muon.py"
click node_checkpoint "https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/checkpoint/checkpointer.py"
click node_normstats "https://github.com/robbyant/lingbot-vla-v2/blob/main/scripts/compute_norm_stats.py"
click node_eval "https://github.com/robbyant/lingbot-vla-v2/blob/main/experiment/robotwin/eval_policy_client_lingbotvla.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_wsclient,node_wsserver,node_policy,node_eval,node_robot toneBlue
class node_model,node_modelcore,node_actionexpert,node_config,node_vision toneAmber
class node_chat,node_multimodal,node_robotdata,node_transform,node_normstats toneMint
class node_trainer,node_parallel,node_parallelstate,node_offload,node_optimizer,node_checkpoint toneRose
```

## Components

### Robot inference

- [Websocket client](https://github.com/robbyant/lingbot-vla-v2/blob/main/deploy/websocket_client_policy.py): `deploy/websocket_client_policy.py`
- [Policy server](https://github.com/robbyant/lingbot-vla-v2/blob/main/deploy/websocket_policy_server.py): `deploy/websocket_policy_server.py`
- [VLA policy](https://github.com/robbyant/lingbot-vla-v2/blob/main/deploy/lingbot_vla_v2_policy.py): `deploy/lingbot_vla_v2_policy.py`
- [RoboTwin evaluator](https://github.com/robbyant/lingbot-vla-v2/blob/main/experiment/robotwin/eval_policy_client_lingbotvla.py): `experiment/robotwin/eval_policy_client_lingbotvla.py`

### VLA modeling

- [LingBot VLA](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla_v2.py): `lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla_v2.py`
- [VLA model core](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla.py): `lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla.py`
- [Action expert](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/models/vla/lingbot_vla/qwen2_action_expert.py): `lingbotvla/models/vla/lingbot_vla/qwen2_action_expert.py`
- [Model config](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/models/vla/lingbot_vla/configuration_lingbot_vla.py): `lingbotvla/models/vla/lingbot_vla/configuration_lingbot_vla.py`
- [Vision teachers](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/models/vla/vision_models/module_utils.py): `lingbotvla/models/vla/vision_models/module_utils.py`

### Data pipeline

- [Chat templates](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/data/chat_template.py): `lingbotvla/data/chat_template.py`
- [Multimodal data](https://github.com/robbyant/lingbot-vla-v2/tree/main/lingbotvla/data/multimodal): `lingbotvla/data/multimodal`
- [Robot datasets](https://github.com/robbyant/lingbot-vla-v2/tree/main/lingbotvla/data/vla_data): `lingbotvla/data/vla_data`
- [Data transforms](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/data/vla_data/transform.py): `lingbotvla/data/vla_data/transform.py`
- [Norm statistics](https://github.com/robbyant/lingbot-vla-v2/blob/main/scripts/compute_norm_stats.py): `scripts/compute_norm_stats.py`

### Training runtime

- [VLA trainer](https://github.com/robbyant/lingbot-vla-v2/blob/main/tasks/vla/train_lingbotvla.py): `tasks/vla/train_lingbotvla.py`
- [Distributed parallelism](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/distributed/torch_parallelize.py): `lingbotvla/distributed/torch_parallelize.py`
- [Parallel state](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/distributed/parallel_state.py): `lingbotvla/distributed/parallel_state.py`
- [Activation offload](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/distributed/offloading.py): `lingbotvla/distributed/offloading.py`
- [Distributed Muon](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/optim/flex_shard/dist_muon.py): `lingbotvla/optim/flex_shard/dist_muon.py`
- [Checkpointing](https://github.com/robbyant/lingbot-vla-v2/blob/main/lingbotvla/checkpoint/checkpointer.py): `lingbotvla/checkpoint/checkpointer.py`

### Other components

- **Robot client**

## Connections

- Robot client → Websocket client: sends observations
- Websocket client → Policy server: requests inference
- Policy server → VLA policy: dispatches policy (evidence: `deploy/lingbot_vla_v2_policy.py`)
- VLA policy → LingBot VLA: loads model (evidence: `deploy/lingbot_vla_v2_policy.py`)
- VLA policy → Model config: uses config (evidence: `deploy/lingbot_vla_v2_policy.py`)
- VLA policy → Robot datasets: uses data utilities (evidence: `deploy/lingbot_vla_v2_policy.py`)
- VLA trainer → LingBot VLA: builds model (evidence: `tasks/vla/train_lingbotvla.py`)
- VLA trainer → Robot datasets: loads data (evidence: `tasks/vla/train_lingbotvla.py`)
- VLA trainer → Distributed parallelism: parallelizes model (evidence: `tasks/vla/train_lingbotvla.py`)
- VLA trainer → Parallel state: initializes state (evidence: `tasks/vla/train_lingbotvla.py`)
- VLA trainer → Activation offload: configures offload (evidence: `tasks/vla/train_lingbotvla.py`)
- VLA trainer → Distributed Muon: uses optimizer (evidence: `tasks/vla/train_lingbotvla.py`)
- VLA trainer → Checkpointing: saves checkpoints (evidence: `tasks/vla/train_lingbotvla.py`)
- VLA trainer → Vision teachers: uses vision teachers (evidence: `tasks/vla/train_lingbotvla.py`)
- VLA trainer → Norm statistics: computes normalization (evidence: `README.md`)
- LingBot VLA → Model config: uses config
- VLA model core → Action expert: uses expert (evidence: `lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla.py`)
- VLA model core → Vision teachers: uses depth head (evidence: `lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla.py`)
- Chat templates → Multimodal data: extends templates (evidence: `lingbotvla/data/multimodal/multimodal_chat_template.py`)
- RoboTwin evaluator → Websocket client: evaluates policy
- Robot datasets → Data transforms: applies transforms