How robbyant/lingbot-va works
LingBot-VA combines autoregressive video-world modeling and robot-action inference, with post-training on LeRobot datasets and standalone or remote inference. The principal evaluation loop sends observations and prompts from a simulator client to a WebSocket policy server, receives action chunks, executes them in the environment, and records outcomes. The model and scheduler form the inference core; training and image-to-video-action generation are separate supported paths. Some server and training wiring is inferred only from README capabilities and sampled excerpts.
Main entry points: wan_va_server.py VA inference runtime · websocket_client_policy.py WebSocket client · train.py Post-training loop · eval_polict_client_openpi.py RoboTwin client
Components 16
Robot Evaluation
- RoboTwin client
evaluation/robotwin/eval_polict_client_openpi.py - LIBERO client
evaluation/libero/client.py - Episodes and videos
evaluation/libero/client.py
Inference Serving
- WebSocket client
wan_va/utils/Simple_Remote_Infer/deploy/websocket_client_policy.py - WebSocket server
wan_va/utils/Simple_Remote_Infer/deploy/websocket_policy_server.py - VA inference runtime
wan_va/wan_va_server.py - Image-to-video-action
wan_va/wan_va_server.py - Model configurations
wan_va/configs/shared_config.py
World-Action Model
- Video-action transformer
wan_va/modules/model.py - Flow-match scheduler
wan_va/utils/scheduler.py
Post-Training
- Latent LeRobot data
wan_va/dataset/lerobot_latent_dataset.py - Post-training loop
wan_va/train.py - Distributed sharding
wan_va/distributed/fsdp.py
Other components
- Robot simulators
- LeRobot datasets
- Robot policy caller
Connections 22
- RoboTwin client → Robot simulators: steps tasks
- LIBERO client → Robot simulators: steps tasks
- RoboTwin client → WebSocket client: sends observations
- LIBERO client → WebSocket client: sends observations
- WebSocket client → WebSocket server: transfers requests
- WebSocket server → VA inference runtime: dispatches inference
- VA inference runtime → Video-action transformer: predicts chunks
- VA inference runtime → Flow-match scheduler: denoises outputs
- VA inference runtime → Model configurations: loads settings
- VA inference runtime → WebSocket server: returns actions
- WebSocket client → RoboTwin client: returns actions
- WebSocket client → LIBERO client: returns actions
- RoboTwin client → Episodes and videos: records episodes
- LIBERO client → Episodes and videos: saves videos
- Robot policy caller → Image-to-video-action: requests generation
- Image-to-video-action → Video-action transformer: generates video-action
- Image-to-video-action → Flow-match scheduler: denoises outputs
- Post-training loop → Latent LeRobot data: loads batches
- Latent LeRobot data → LeRobot datasets: reads episodes
- Post-training loop → Video-action transformer: trains model
- Post-training loop → Flow-match scheduler: adds training noise
- Post-training loop → Distributed sharding: shards model