How open-gigaai/giga-world-policy works
Purpose and Entry Points
GigaWorld-Policy is a PyTorch World Action Model (WAM) training and inference repository for robot control. It trains with action and future-visual-dynamics supervision but serves action predictions without explicit future-video generation. Main entry points are scripts/train.py, scripts/inference_openloop.py, and scripts/inference_openloop_gwp0.py; shell launchers establish inference server/client roles in scripts/run_inference_openloop.sh and scripts/run_inference_openloop_gwp0.sh.
Configuration-Driven Variants
Top-level Python configurations select the GWP-0.5 Mixture-of-Transformers variant or GWP-0 baseline and bind dataset locations, pretrained WAN assets, normalization, checkpoints, and output projects through environment variables. Paths: configs/giga_world_policy_0_5_agilex_finetune.py, configs/giga_world_policy_0_agilex_finetune.py, scripts/train.py.
Data Preparation and Dataset Boundary
Input is a LeRobot v3 dataset augmented before training with action/task normalization statistics and cached T5 prompt embeddings. The WAM-specific dataset adapter consumes this prepared data, while the vendored dataset package supplies storage-format implementations including LeRobot, LMDB, and PKL. Paths: scripts/compute_wam_task_norm.py, scripts/compute_t5_embedding.py, world_action_model/datasets/wam_lerobot_dataset.py.
WAM Model Stack
The core model family extends WAN transformer components with action-conditioning and specialized action projection. GWP-0.5’s causal Mixture-of-Transformers implementation separates visual-dynamics and action-generation computation; the baseline and alternate causal/MoT forms remain selectable. Paths: world_action_model/models/transformer_wa_casual_mot.py, world_action_model/models/transformer_wa_casual.py, world_action_model/models/action_projectors.py.
Training Pipeline
Training builds the configured model, transforms batches into WAM representations, and uses dedicated causal trainers for finetuning, MoT training, or pretraining. The vendored giga-train layer supplies generic trainer, optimizer/scheduler construction, registries, and distributed execution. Paths: world_action_model/trainer/wa_casual_trainer_mot.py, world_action_model/transforms/wa_transforms.py, third_party/giga-train/giga_train/trainers/trainer.py.
Base Video-Model Dependencies
WAN is a vendored model/runtime dependency providing transformer architecture, VAE, T5 text encoder/tokenizers, diffusion/flow-matching utilities, and distributed primitives. It is used for pretrained initialization and training-side visual modeling rather than being the primary action-only inference path. Paths: third_party/wan/modules/model.py, third_party/wan/modules/t5.py, third_party/wan/modules/vae2_2.py.
Inference and Deployment Boundary
Open-loop Python inference is explicitly split into a socket server holding model state and a client issuing requests, using the vendored socket transport. It loads a trained transformer checkpoint, WAN base-model assets, and normalization statistics. For local low-latency deployment, the external wam.cpp C++/CUDA runtime converts checkpoints to GGUF and remains wire-compatible with this client protocol. Paths: scripts/inference_openloop.py, third_party/giga-models/giga_models/sockets/server.py, third_party/giga-models/giga_models/sockets/client.py.
Runtime and External Services
The primary runtime is Python 3.11 with CUDA PyTorch 2.7.1; distributed training supports Accelerate/DDP and DeepSpeed ZeRO configurations. Required external artifacts/services are LeRobot datasets, WAN pretrained weights, Hugging Face-hosted GWP checkpoints, and optional Hugging Face CLI/API download. Paths: requirements.txt, third_party/giga-train/giga_train/distributed/launch.py, third_party/giga-train/giga_train/distributed/accelerate_configs/config_deepspeed_zero2.json.
Main entry points: wa_casual_trainer_mot.py Causal MoT trainer · model.py WAN transformer runtime · wam_lerobot_dataset.py WAM LeRobot adapter · train.py Training entry point
Components 21
Data preparation
- LeRobot v3 dataset
- Task/action normalization
scripts/compute_wam_task_norm.py - T5 prompt embedding cache
scripts/compute_t5_embedding.py - WAM LeRobot adapter · Consumes augmented data as WAM training samples.
world_action_model/datasets/wam_lerobot_dataset.py
Training
- GWP-0.5 config · Selects the Mixture-of-Transformers finetuning variant.
configs/giga_world_policy_0_5_agilex_finetune.py - GWP-0 config · Selects the baseline variant.
configs/giga_world_policy_0_agilex_finetune.py - Training entry point
scripts/train.py - WAM batch transforms
world_action_model/transforms/wa_transforms.py - Causal MoT trainer · Runs MoT WAM finetuning and training logic.
world_action_model/trainer/wa_casual_trainer_mot.py - giga-train framework · Provides generic training, optimization, registries, and distributed execution.
third_party/giga-train/giga_train/trainers/trainer.py - Distributed launch · Supports Accelerate, DDP, and DeepSpeed execution.
third_party/giga-train/giga_train/distributed/launch.py
WAM model stack
- Causal MoT transformer · Separates visual-dynamics and action-generation computation.
world_action_model/models/transformer_wa_casual_mot.py - Causal baseline transformer
world_action_model/models/transformer_wa_casual.py - Action projectors · Projects action representations for WAM conditioning and prediction.
world_action_model/models/action_projectors.py
Inference deployment
- GWP checkpoint · Trained transformer weights, optionally obtained from Hugging Face.
- Open-loop inference · Holds model state and serves action predictions without future-video generation.
scripts/inference_openloop.py - Socket server
third_party/giga-models/giga_models/sockets/server.py - Socket client · Issues action-prediction requests to the inference server.
third_party/giga-models/giga_models/sockets/client.py - wam.cpp runtime · Converts checkpoints to GGUF and is wire-compatible with the client protocol.
WAN dependency
- WAN transformer runtime · Supplies pretrained initialization and training-side visual modeling.
third_party/wan/modules/model.py - WAN text and VAE assets · Provides T5 encoding and VAE components.
third_party/wan/modules/t5.py
Connections 23
- LeRobot v3 dataset → Task/action normalization: normalize
- LeRobot v3 dataset → T5 prompt embedding cache: embed prompts
- Task/action normalization → WAM LeRobot adapter: augmented data
- T5 prompt embedding cache → WAM LeRobot adapter: embedding cache
- GWP-0.5 config → Training entry point: configures
- GWP-0 config → Training entry point: configures
- WAM LeRobot adapter → WAM batch transforms: batches
- Training entry point → WAM batch transforms: builds pipeline
- WAM batch transforms → Causal MoT trainer: WAM representations
- Causal MoT trainer → giga-train framework: uses framework
- giga-train framework → Distributed launch: launches distributed jobs
- Causal MoT trainer → Causal MoT transformer: trains
- Causal MoT transformer → Action projectors: predicts through
- Causal baseline transformer → Action projectors: uses
- WAN transformer runtime → Causal MoT transformer: base initialization
- WAN text and VAE assets → WAN transformer runtime: supports
- Causal MoT trainer → GWP checkpoint: writes
- GWP checkpoint → Open-loop inference: loads
- WAN transformer runtime → Open-loop inference: loads base assets
- Open-loop inference → Socket server: serves via
- Socket client → Socket server: requests
- Socket client → wam.cpp runtime: compatible protocol
- GWP checkpoint → wam.cpp runtime: converts to GGUF