# robbyant/lingbot-world-v2 architecture

> Architecture diagram of the GitHub repository robbyant/lingbot-world-v2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/robbyant/lingbot-world-v2
- Repository: https://github.com/robbyant/lingbot-world-v2

## Overview

#### Purpose and Entry Points
LingBot-World 2.0 is inference code for a 14B causal-fast interactive image-to-video world model, built on Wan2.2. It extends an input image into long video sequences conditioned by text and recorded interaction/camera controls. Primary CLI entry point: `generate.py`; convenience launcher: `run_fast.sh`. The public repository provides offline distributed generation, not the real-time deployment/agent harness.

#### Inference Orchestration
`generate.py` parses task, checkpoint, image, prompt, action directory, resolution, frame count, and distributed flags. It selects the image-to-video pipeline and runs chunked causal generation with KV caching, avoiding generation of the full sequence in one pass. Model/task defaults are defined under `wan/configs/wan_i2v_A14B.py` and shared in `wan/configs/shared_config.py`.

#### Image-to-Video Pipeline Boundary
`wan/image2video.py` is the high-level pipeline boundary: it accepts user conditioning and model paths, prepares image/text/control inputs, iteratively samples video chunks, and emits decoded output. It bridges application-facing inputs from `generate.py` to neural modules in `wan/modules/`, while camera/action transformations are supplied by `wan/utils/cam_utils.py`.

#### Causal Video Backbone
The denoising transformer family lives in `wan/modules/model.py`, `wan/modules/model_causal.py`, and `wan/modules/model_fast.py`. The causal variants preserve temporal state/KV cache across chunks, enabling an unbounded interaction horizon; the fast variant is the released distilled few-step model. Attention implementation and sequence behavior are concentrated in `wan/modules/attention.py`.

#### Conditioning and Latent Video Representation
Text prompts are tokenized/encoded through `wan/modules/tokenizers.py` and `wan/modules/t5.py`. The input image and generated latent video are encoded/decoded by `wan/modules/vae2_1.py`. This establishes the core conditioning path: prompt + initial image + interaction/camera trajectories → latent denoiser → decoded frames.

#### Sampling and Interaction Inputs
Diffusion/flow sampling is implemented in `wan/utils/fm_solvers.py` and `wan/utils/fm_solvers_unipc.py`. Action datasets are filesystem-based example bundles such as `examples/03/action.npy`, `examples/03/poses.npy`, and `examples/03/intrinsics.npy`; some scenarios additionally provide keyboard-specific action arrays and prompts. These controls are passed via `--action_path`, forming an explicit boundary between externally authored trajectories and learned video synthesis.

#### Distributed Runtime and Dependencies
The intended runtime is PyTorch 2.4+ with FlashAttention, launched via `torchrun` on multi-GPU hardware. FSDP wrapping and distributed mechanics are in `wan/distributed/fsdp.py` and `wan/distributed/util.py`; Ulysses/sequence parallelism is implemented in `wan/distributed/ulysses.py` and `wan/distributed/sequence_parallel.py`. Checkpoint weights are external downloads from Hugging Face or ModelScope; deployment is explicitly delegated to external SGLang or NVIDIA flashdreams integrations.

Main entry points: [`wan/modules/model_causal.py`](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/model_causal.py) (Causal denoiser), [`wan/image2video.py`](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/image2video.py) (Image-to-video pipeline), [`wan/modules/vae2_1.py`](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/vae2_1.py) (Video VAE), [`generate.py`](https://github.com/Robbyant/lingbot-world-v2/blob/main/generate.py) (Generation CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Generation Entry"]
  node_launcher["Convenience launcher<br/>shell launcher<br/>[run_fast.sh]"]
  node_cli["Generation CLI<br/>orchestrator<br/>[generate.py]"]
  node_config["I2V defaults<br/>model config<br/>[wan_i2v_A14B.py]"]
  node_shared_config["Shared defaults<br/>config<br/>[shared_config.py]"]
end

subgraph group_pipeline["I2V Pipeline"]
  node_controls["Action and camera bundles<br/>external input<br/>[action.npy]"]
  node_camera_utils["Camera/action transforms<br/>conditioning utility<br/>[cam_utils.py]"]
  node_i2v_pipeline["Image-to-video pipeline<br/>pipeline boundary<br/>[image2video.py]"]
end

subgraph group_model["Neural Generation"]
  node_tokenizers["Prompt tokenization<br/>text preprocessing<br/>[tokenizers.py]"]
  node_t5["Text encoder<br/>T5 encoder<br/>[t5.py]"]
  node_vae["Video VAE<br/>latent codec<br/>[vae2_1.py]"]
  node_causal_backbone["Causal denoiser<br/>video transformer<br/>[model_causal.py]"]
  node_fast_backbone["Fast denoiser<br/>distilled transformer<br/>[model_fast.py]"]
  node_attention["Attention and KV cache<br/>attention layer<br/>[attention.py]"]
  node_sampler["Flow sampler<br/>diffusion solver<br/>[fm_solvers.py]"]
end

subgraph group_runtime["Distributed Runtime"]
  node_fsdp["FSDP wrapping<br/>distributed execution<br/>[fsdp.py]"]
  node_sequence_parallel["Sequence parallelism<br/>distributed execution"]
end

node_checkpoints[("External checkpoints<br/>model weights")]
node_video_output["Generated video<br/>output"]

node_launcher -->|"launches"| node_cli
node_cli -->|"loads defaults"| node_config
node_config -->|"shares settings"| node_shared_config
node_cli -->|"passes task inputs"| node_i2v_pipeline
node_controls -->|"actions and poses"| node_camera_utils
node_camera_utils -->|"control trajectories"| node_i2v_pipeline
node_i2v_pipeline -->|"prompt"| node_tokenizers
node_tokenizers -->|"tokens"| node_t5
node_t5 -->|"text conditioning"| node_causal_backbone
node_i2v_pipeline -->|"input image / latents"| node_vae
node_vae -->|"image and video latents"| node_causal_backbone
node_i2v_pipeline -->|"chunk sampling"| node_sampler
node_sampler -->|"denoising steps"| node_causal_backbone
node_causal_backbone -->|"temporal attention"| node_attention
node_attention -->|"KV cache across chunks"| node_causal_backbone
node_fast_backbone -->|"uses"| node_attention
node_checkpoints -.->|"weights"| node_causal_backbone
node_checkpoints -.->|"weights"| node_fast_backbone
node_causal_backbone -->|"generated latents"| node_vae
node_vae -->|"decoded frames"| node_video_output
node_fsdp -->|"wraps model"| node_causal_backbone
node_sequence_parallel -->|"partitions sequences"| node_causal_backbone

click node_launcher "https://github.com/Robbyant/lingbot-world-v2/blob/main/run_fast.sh"
click node_cli "https://github.com/Robbyant/lingbot-world-v2/blob/main/generate.py"
click node_config "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/configs/wan_i2v_A14B.py"
click node_shared_config "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/configs/shared_config.py"
click node_controls "https://github.com/Robbyant/lingbot-world-v2/blob/main/examples/03/action.npy"
click node_camera_utils "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/utils/cam_utils.py"
click node_i2v_pipeline "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/image2video.py"
click node_tokenizers "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/tokenizers.py"
click node_t5 "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/t5.py"
click node_vae "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/vae2_1.py"
click node_causal_backbone "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/model_causal.py"
click node_fast_backbone "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/model_fast.py"
click node_attention "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/attention.py"
click node_sampler "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/utils/fm_solvers.py"
click node_fsdp "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/distributed/fsdp.py"
click node_sequence_parallel "https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/distributed/sequence_parallel.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_launcher,node_cli,node_config,node_shared_config toneBlue
class node_controls,node_camera_utils,node_i2v_pipeline toneAmber
class node_tokenizers,node_t5,node_vae,node_causal_backbone,node_fast_backbone,node_attention,node_sampler toneMint
class node_fsdp,node_sequence_parallel toneRose
class node_checkpoints,node_video_output toneNeutral
```

## Components

### Generation Entry

- [Convenience launcher](https://github.com/Robbyant/lingbot-world-v2/blob/main/run_fast.sh): shell launcher · `run_fast.sh`
- [Generation CLI](https://github.com/Robbyant/lingbot-world-v2/blob/main/generate.py): orchestrator · Parses inputs and runs distributed chunked generation. · `generate.py`
- [I2V defaults](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/configs/wan_i2v_A14B.py): model config · `wan/configs/wan_i2v_A14B.py`
- [Shared defaults](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/configs/shared_config.py): config · `wan/configs/shared_config.py`

### I2V Pipeline

- [Action and camera bundles](https://github.com/Robbyant/lingbot-world-v2/blob/main/examples/03/action.npy): external input · Filesystem trajectories supplied through --action_path. · `examples/03/action.npy`
- [Camera/action transforms](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/utils/cam_utils.py): conditioning utility · `wan/utils/cam_utils.py`
- [Image-to-video pipeline](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/image2video.py): pipeline boundary · Prepares conditioning, samples chunks, and emits decoded video. · `wan/image2video.py`

### Neural Generation

- [Prompt tokenization](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/tokenizers.py): text preprocessing · `wan/modules/tokenizers.py`
- [Text encoder](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/t5.py): T5 encoder · `wan/modules/t5.py`
- [Video VAE](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/vae2_1.py): latent codec · Encodes the initial image and decodes generated latent video. · `wan/modules/vae2_1.py`
- [Causal denoiser](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/model_causal.py): video transformer · Preserves temporal state and KV cache between chunks. · `wan/modules/model_causal.py`
- [Fast denoiser](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/model_fast.py): distilled transformer · Released few-step causal model variant. · `wan/modules/model_fast.py`
- [Attention and KV cache](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/modules/attention.py): attention layer · `wan/modules/attention.py`
- [Flow sampler](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/utils/fm_solvers.py): diffusion solver · `wan/utils/fm_solvers.py`

### Distributed Runtime

- [FSDP wrapping](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/distributed/fsdp.py): distributed execution · `wan/distributed/fsdp.py`
- [Sequence parallelism](https://github.com/Robbyant/lingbot-world-v2/blob/main/wan/distributed/sequence_parallel.py): distributed execution · Ulysses-style sequence-parallel execution. · `wan/distributed/sequence_parallel.py`

### Other components

- **External checkpoints**: model weights · Downloaded from Hugging Face or ModelScope.
- **Generated video**: output

## Connections

- Convenience launcher → Generation CLI: launches
- Generation CLI → I2V defaults: loads defaults
- I2V defaults → Shared defaults: shares settings
- Generation CLI → Image-to-video pipeline: passes task inputs
- Action and camera bundles → Camera/action transforms: actions and poses
- Camera/action transforms → Image-to-video pipeline: control trajectories
- Image-to-video pipeline → Prompt tokenization: prompt
- Prompt tokenization → Text encoder: tokens
- Text encoder → Causal denoiser: text conditioning
- Image-to-video pipeline → Video VAE: input image / latents
- Video VAE → Causal denoiser: image and video latents
- Image-to-video pipeline → Flow sampler: chunk sampling
- Flow sampler → Causal denoiser: denoising steps
- Causal denoiser → Attention and KV cache: temporal attention
- Attention and KV cache → Causal denoiser: KV cache across chunks
- Fast denoiser → Attention and KV cache: uses
- External checkpoints → Causal denoiser: weights
- External checkpoints → Fast denoiser: weights
- Causal denoiser → Video VAE: generated latents
- Video VAE → Generated video: decoded frames
- FSDP wrapping → Causal denoiser: wraps model
- Sequence parallelism → Causal denoiser: partitions sequences