How robbyant/lingbot-vla-v2 works
LingBot-VLA 2.0 is a vision-language-action foundation model and robot policy stack. The main path is robot observations sent through a websocket client to a policy server, where the model produces actions for the robot; the server-side handoff is not verified in the supplied excerpts. The repository also supports heterogeneous robot/video training, multimodal data preparation, predictive depth and video supervision, and distributed post-training. Training and model links are grounded in the source index; unsampled runtime links are omitted or marked unverified.
Main entry points: train_lingbotvla.py VLA trainer · lingbot_vla_v2_policy.py VLA policy · websocket_client_policy.py Websocket client · modeling_lingbot_vla_v2.py LingBot VLA
Components 21
Robot inference
- Websocket client
deploy/websocket_client_policy.py - Policy server
deploy/websocket_policy_server.py - VLA policy
deploy/lingbot_vla_v2_policy.py - RoboTwin evaluator
experiment/robotwin/eval_policy_client_lingbotvla.py
VLA modeling
- LingBot VLA
lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla_v2.py - VLA model core
lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla.py - Action expert
lingbotvla/models/vla/lingbot_vla/qwen2_action_expert.py - Model config
lingbotvla/models/vla/lingbot_vla/configuration_lingbot_vla.py - Vision teachers
lingbotvla/models/vla/vision_models/module_utils.py
Data pipeline
- Chat templates
lingbotvla/data/chat_template.py - Multimodal data
lingbotvla/data/multimodal - Robot datasets
lingbotvla/data/vla_data - Data transforms
lingbotvla/data/vla_data/transform.py - Norm statistics
scripts/compute_norm_stats.py
Training runtime
- VLA trainer
tasks/vla/train_lingbotvla.py - Distributed parallelism
lingbotvla/distributed/torch_parallelize.py - Parallel state
lingbotvla/distributed/parallel_state.py - Activation offload
lingbotvla/distributed/offloading.py - Distributed Muon
lingbotvla/optim/flex_shard/dist_muon.py - Checkpointing
lingbotvla/checkpoint/checkpointer.py
Other components
- Robot client
Connections 21
- Robot client → Websocket client: sends observations
- Websocket client → Policy server: requests inference
- Policy server → VLA policy: dispatches policy
deploy/lingbot_vla_v2_policy.py - VLA policy → LingBot VLA: loads model
deploy/lingbot_vla_v2_policy.py - VLA policy → Model config: uses config
deploy/lingbot_vla_v2_policy.py - VLA policy → Robot datasets: uses data utilities
deploy/lingbot_vla_v2_policy.py - VLA trainer → LingBot VLA: builds model
tasks/vla/train_lingbotvla.py - VLA trainer → Robot datasets: loads data
tasks/vla/train_lingbotvla.py - VLA trainer → Distributed parallelism: parallelizes model
tasks/vla/train_lingbotvla.py - VLA trainer → Parallel state: initializes state
tasks/vla/train_lingbotvla.py - VLA trainer → Activation offload: configures offload
tasks/vla/train_lingbotvla.py - VLA trainer → Distributed Muon: uses optimizer
tasks/vla/train_lingbotvla.py - VLA trainer → Checkpointing: saves checkpoints
tasks/vla/train_lingbotvla.py - VLA trainer → Vision teachers: uses vision teachers
tasks/vla/train_lingbotvla.py - VLA trainer → Norm statistics: computes normalization
README.md - LingBot VLA → Model config: uses config
- VLA model core → Action expert: uses expert
lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla.py - VLA model core → Vision teachers: uses depth head
lingbotvla/models/vla/lingbot_vla/modeling_lingbot_vla.py - Chat templates → Multimodal data: extends templates
lingbotvla/data/multimodal/multimodal_chat_template.py - RoboTwin evaluator → Websocket client: evaluates policy
- Robot datasets → Data transforms: applies transforms