# open-gigaai/giga-world-policy architecture

> Architecture diagram of the GitHub repository open-gigaai/giga-world-policy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/open-gigaai/giga-world-policy
- Repository: https://github.com/open-gigaai/giga-world-policy

## Overview

#### Purpose and Entry Points
GigaWorld-Policy is a PyTorch World Action Model (WAM) training and inference repository for robot control. It trains with action and future-visual-dynamics supervision but serves action predictions without explicit future-video generation. Main entry points are `scripts/train.py`, `scripts/inference_openloop.py`, and `scripts/inference_openloop_gwp0.py`; shell launchers establish inference server/client roles in `scripts/run_inference_openloop.sh` and `scripts/run_inference_openloop_gwp0.sh`.

#### Configuration-Driven Variants
Top-level Python configurations select the GWP-0.5 Mixture-of-Transformers variant or GWP-0 baseline and bind dataset locations, pretrained WAN assets, normalization, checkpoints, and output projects through environment variables. Paths: `configs/giga_world_policy_0_5_agilex_finetune.py`, `configs/giga_world_policy_0_agilex_finetune.py`, `scripts/train.py`.

#### Data Preparation and Dataset Boundary
Input is a LeRobot v3 dataset augmented before training with action/task normalization statistics and cached T5 prompt embeddings. The WAM-specific dataset adapter consumes this prepared data, while the vendored dataset package supplies storage-format implementations including LeRobot, LMDB, and PKL. Paths: `scripts/compute_wam_task_norm.py`, `scripts/compute_t5_embedding.py`, `world_action_model/datasets/wam_lerobot_dataset.py`.

#### WAM Model Stack
The core model family extends WAN transformer components with action-conditioning and specialized action projection. GWP-0.5’s causal Mixture-of-Transformers implementation separates visual-dynamics and action-generation computation; the baseline and alternate causal/MoT forms remain selectable. Paths: `world_action_model/models/transformer_wa_casual_mot.py`, `world_action_model/models/transformer_wa_casual.py`, `world_action_model/models/action_projectors.py`.

#### Training Pipeline
Training builds the configured model, transforms batches into WAM representations, and uses dedicated causal trainers for finetuning, MoT training, or pretraining. The vendored `giga-train` layer supplies generic trainer, optimizer/scheduler construction, registries, and distributed execution. Paths: `world_action_model/trainer/wa_casual_trainer_mot.py`, `world_action_model/transforms/wa_transforms.py`, `third_party/giga-train/giga_train/trainers/trainer.py`.

#### Base Video-Model Dependencies
WAN is a vendored model/runtime dependency providing transformer architecture, VAE, T5 text encoder/tokenizers, diffusion/flow-matching utilities, and distributed primitives. It is used for pretrained initialization and training-side visual modeling rather than being the primary action-only inference path. Paths: `third_party/wan/modules/model.py`, `third_party/wan/modules/t5.py`, `third_party/wan/modules/vae2_2.py`.

#### Inference and Deployment Boundary
Open-loop Python inference is explicitly split into a socket server holding model state and a client issuing requests, using the vendored socket transport. It loads a trained transformer checkpoint, WAN base-model assets, and normalization statistics. For local low-latency deployment, the external `wam.cpp` C++/CUDA runtime converts checkpoints to GGUF and remains wire-compatible with this client protocol. Paths: `scripts/inference_openloop.py`, `third_party/giga-models/giga_models/sockets/server.py`, `third_party/giga-models/giga_models/sockets/client.py`.

#### Runtime and External Services
The primary runtime is Python 3.11 with CUDA PyTorch 2.7.1; distributed training supports Accelerate/DDP and DeepSpeed ZeRO configurations. Required external artifacts/services are LeRobot datasets, WAN pretrained weights, Hugging Face-hosted GWP checkpoints, and optional Hugging Face CLI/API download. Paths: `requirements.txt`, `third_party/giga-train/giga_train/distributed/launch.py`, `third_party/giga-train/giga_train/distributed/accelerate_configs/config_deepspeed_zero2.json`.

Main entry points: [`world_action_model/trainer/wa_casual_trainer_mot.py`](https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/trainer/wa_casual_trainer_mot.py) (Causal MoT trainer), [`third_party/wan/modules/model.py`](https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/wan/modules/model.py) (WAN transformer runtime), [`world_action_model/datasets/wam_lerobot_dataset.py`](https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/datasets/wam_lerobot_dataset.py) (WAM LeRobot adapter), [`scripts/train.py`](https://github.com/open-gigaai/giga-world-policy/blob/main/scripts/train.py) (Training entry point)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Data preparation"]
  node_raw_dataset[("LeRobot v3 dataset<br/>external dataset")]
  node_norm_prep["Task/action normalization<br/>preparation CLI"]
  node_prompt_prep["T5 prompt embedding cache<br/>preparation CLI"]
  node_wam_dataset["WAM LeRobot adapter<br/>dataset adapter"]
end

subgraph group_training["Training"]
  node_config05["GWP-0.5 config<br/>Python config"]
  node_config0["GWP-0 config<br/>Python config"]
  node_train_entry["Training entry point<br/>Python CLI<br/>[train.py]"]
  node_transforms["WAM batch transforms<br/>training transforms<br/>[wa_transforms.py]"]
  node_mot_trainer["Causal MoT trainer<br/>specialized trainer"]
  node_giga_train["giga-train framework<br/>training framework<br/>[trainer.py]"]
  node_distributed["Distributed launch<br/>distributed runtime<br/>[launch.py]"]
end

subgraph group_model["WAM model stack"]
  node_mot_transformer["Causal MoT transformer<br/>WAM transformer"]
  node_causal_transformer["Causal baseline transformer<br/>WAM transformer"]
  node_action_projectors["Action projectors<br/>model component"]
end

subgraph group_inference["Inference deployment"]
  node_trained_checkpoint["GWP checkpoint<br/>model artifact"]
  node_inference_entry["Open-loop inference<br/>Python server CLI"]
  node_socket_server["Socket server<br/>transport server<br/>[server.py]"]
  node_socket_client["Socket client<br/>transport client<br/>[client.py]"]
  node_wam_cpp["wam.cpp runtime<br/>external C++/CUDA runtime"]
end

subgraph group_wan["WAN dependency"]
  node_wan_model["WAN transformer runtime<br/>vendored video model<br/>[model.py]"]
  node_wan_text_vae["WAN text and VAE assets<br/>vendored model assets<br/>[t5.py]"]
end

node_raw_dataset -->|"normalize"| node_norm_prep
node_raw_dataset -->|"embed prompts"| node_prompt_prep
node_norm_prep -->|"augmented data"| node_wam_dataset
node_prompt_prep -->|"embedding cache"| node_wam_dataset
node_config05 -->|"configures"| node_train_entry
node_config0 -->|"configures"| node_train_entry
node_wam_dataset -->|"batches"| node_transforms
node_train_entry -->|"builds pipeline"| node_transforms
node_transforms -->|"WAM representations"| node_mot_trainer
node_mot_trainer -->|"uses framework"| node_giga_train
node_giga_train -->|"launches distributed jobs"| node_distributed
node_mot_trainer -->|"trains"| node_mot_transformer
node_mot_transformer -->|"predicts through"| node_action_projectors
node_causal_transformer -->|"uses"| node_action_projectors
node_wan_model -->|"base initialization"| node_mot_transformer
node_wan_text_vae -->|"supports"| node_wan_model
node_mot_trainer -->|"writes"| node_trained_checkpoint
node_trained_checkpoint -->|"loads"| node_inference_entry
node_wan_model -.->|"loads base assets"| node_inference_entry
node_inference_entry -->|"serves via"| node_socket_server
node_socket_client -->|"requests"| node_socket_server
node_socket_client -.->|"compatible protocol"| node_wam_cpp
node_trained_checkpoint -->|"converts to GGUF"| node_wam_cpp

click node_config05 "https://github.com/open-gigaai/giga-world-policy/blob/main/configs/giga_world_policy_0_5_agilex_finetune.py"
click node_config0 "https://github.com/open-gigaai/giga-world-policy/blob/main/configs/giga_world_policy_0_agilex_finetune.py"
click node_train_entry "https://github.com/open-gigaai/giga-world-policy/blob/main/scripts/train.py"
click node_norm_prep "https://github.com/open-gigaai/giga-world-policy/blob/main/scripts/compute_wam_task_norm.py"
click node_prompt_prep "https://github.com/open-gigaai/giga-world-policy/blob/main/scripts/compute_t5_embedding.py"
click node_wam_dataset "https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/datasets/wam_lerobot_dataset.py"
click node_transforms "https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/transforms/wa_transforms.py"
click node_mot_trainer "https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/trainer/wa_casual_trainer_mot.py"
click node_giga_train "https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/giga-train/giga_train/trainers/trainer.py"
click node_distributed "https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/giga-train/giga_train/distributed/launch.py"
click node_mot_transformer "https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/models/transformer_wa_casual_mot.py"
click node_causal_transformer "https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/models/transformer_wa_casual.py"
click node_action_projectors "https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/models/action_projectors.py"
click node_wan_model "https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/wan/modules/model.py"
click node_wan_text_vae "https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/wan/modules/t5.py"
click node_inference_entry "https://github.com/open-gigaai/giga-world-policy/blob/main/scripts/inference_openloop.py"
click node_socket_server "https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/giga-models/giga_models/sockets/server.py"
click node_socket_client "https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/giga-models/giga_models/sockets/client.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_dataset,node_norm_prep,node_prompt_prep,node_wam_dataset toneBlue
class node_config05,node_config0,node_train_entry,node_transforms,node_mot_trainer,node_giga_train,node_distributed toneAmber
class node_mot_transformer,node_causal_transformer,node_action_projectors toneMint
class node_trained_checkpoint,node_inference_entry,node_socket_server,node_socket_client,node_wam_cpp toneRose
class node_wan_model,node_wan_text_vae toneIndigo
```

## Components

### Data preparation

- **LeRobot v3 dataset**: external dataset
- [Task/action normalization](https://github.com/open-gigaai/giga-world-policy/blob/main/scripts/compute_wam_task_norm.py): preparation CLI · `scripts/compute_wam_task_norm.py`
- [T5 prompt embedding cache](https://github.com/open-gigaai/giga-world-policy/blob/main/scripts/compute_t5_embedding.py): preparation CLI · `scripts/compute_t5_embedding.py`
- [WAM LeRobot adapter](https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/datasets/wam_lerobot_dataset.py): dataset adapter · Consumes augmented data as WAM training samples. · `world_action_model/datasets/wam_lerobot_dataset.py`

### Training

- [GWP-0.5 config](https://github.com/open-gigaai/giga-world-policy/blob/main/configs/giga_world_policy_0_5_agilex_finetune.py): Python config · Selects the Mixture-of-Transformers finetuning variant. · `configs/giga_world_policy_0_5_agilex_finetune.py`
- [GWP-0 config](https://github.com/open-gigaai/giga-world-policy/blob/main/configs/giga_world_policy_0_agilex_finetune.py): Python config · Selects the baseline variant. · `configs/giga_world_policy_0_agilex_finetune.py`
- [Training entry point](https://github.com/open-gigaai/giga-world-policy/blob/main/scripts/train.py): Python CLI · `scripts/train.py`
- [WAM batch transforms](https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/transforms/wa_transforms.py): training transforms · `world_action_model/transforms/wa_transforms.py`
- [Causal MoT trainer](https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/trainer/wa_casual_trainer_mot.py): specialized trainer · Runs MoT WAM finetuning and training logic. · `world_action_model/trainer/wa_casual_trainer_mot.py`
- [giga-train framework](https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/giga-train/giga_train/trainers/trainer.py): training framework · Provides generic training, optimization, registries, and distributed execution. · `third_party/giga-train/giga_train/trainers/trainer.py`
- [Distributed launch](https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/giga-train/giga_train/distributed/launch.py): distributed runtime · Supports Accelerate, DDP, and DeepSpeed execution. · `third_party/giga-train/giga_train/distributed/launch.py`

### WAM model stack

- [Causal MoT transformer](https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/models/transformer_wa_casual_mot.py): WAM transformer · Separates visual-dynamics and action-generation computation. · `world_action_model/models/transformer_wa_casual_mot.py`
- [Causal baseline transformer](https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/models/transformer_wa_casual.py): WAM transformer · `world_action_model/models/transformer_wa_casual.py`
- [Action projectors](https://github.com/open-gigaai/giga-world-policy/blob/main/world_action_model/models/action_projectors.py): model component · Projects action representations for WAM conditioning and prediction. · `world_action_model/models/action_projectors.py`

### Inference deployment

- **GWP checkpoint**: model artifact · Trained transformer weights, optionally obtained from Hugging Face.
- [Open-loop inference](https://github.com/open-gigaai/giga-world-policy/blob/main/scripts/inference_openloop.py): Python server CLI · Holds model state and serves action predictions without future-video generation. · `scripts/inference_openloop.py`
- [Socket server](https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/giga-models/giga_models/sockets/server.py): transport server · `third_party/giga-models/giga_models/sockets/server.py`
- [Socket client](https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/giga-models/giga_models/sockets/client.py): transport client · Issues action-prediction requests to the inference server. · `third_party/giga-models/giga_models/sockets/client.py`
- **wam.cpp runtime**: external C++/CUDA runtime · Converts checkpoints to GGUF and is wire-compatible with the client protocol.

### WAN dependency

- [WAN transformer runtime](https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/wan/modules/model.py): vendored video model · Supplies pretrained initialization and training-side visual modeling. · `third_party/wan/modules/model.py`
- [WAN text and VAE assets](https://github.com/open-gigaai/giga-world-policy/blob/main/third_party/wan/modules/t5.py): vendored model assets · Provides T5 encoding and VAE components. · `third_party/wan/modules/t5.py`

## Connections

- LeRobot v3 dataset → Task/action normalization: normalize
- LeRobot v3 dataset → T5 prompt embedding cache: embed prompts
- Task/action normalization → WAM LeRobot adapter: augmented data
- T5 prompt embedding cache → WAM LeRobot adapter: embedding cache
- GWP-0.5 config → Training entry point: configures
- GWP-0 config → Training entry point: configures
- WAM LeRobot adapter → WAM batch transforms: batches
- Training entry point → WAM batch transforms: builds pipeline
- WAM batch transforms → Causal MoT trainer: WAM representations
- Causal MoT trainer → giga-train framework: uses framework
- giga-train framework → Distributed launch: launches distributed jobs
- Causal MoT trainer → Causal MoT transformer: trains
- Causal MoT transformer → Action projectors: predicts through
- Causal baseline transformer → Action projectors: uses
- WAN transformer runtime → Causal MoT transformer: base initialization
- WAN text and VAE assets → WAN transformer runtime: supports
- Causal MoT trainer → GWP checkpoint: writes
- GWP checkpoint → Open-loop inference: loads
- WAN transformer runtime → Open-loop inference: loads base assets
- Open-loop inference → Socket server: serves via
- Socket client → Socket server: requests
- Socket client → wam.cpp runtime: compatible protocol
- GWP checkpoint → wam.cpp runtime: converts to GGUF