robbyant/lingbot-world-v2

Purpose and Entry Points

LingBot-World 2.0 is inference code for a 14B causal-fast interactive image-to-video world model, built on Wan2.2. It extends an input image into long video sequences conditioned by text and recorded interaction/camera controls. Primary CLI entry point: generate.py; convenience launcher: run_fast.sh. The public repository provides offline distributed generation, not the real-time deployment/agent harness.

Inference Orchestration

generate.py parses task, checkpoint, image, prompt, action directory, resolution, frame count, and distributed flags. It selects the image-to-video pipeline and runs chunked causal generation with KV caching, avoiding generation of the full sequence in one pass. Model/task defaults are defined under wan/configs/wan_i2v_A14B.py and shared in wan/configs/shared_config.py.

Image-to-Video Pipeline Boundary

wan/image2video.py is the high-level pipeline boundary: it accepts user conditioning and model paths, prepares image/text/control inputs, iteratively samples video chunks, and emits decoded output. It bridges application-facing inputs from generate.py to neural modules in wan/modules/, while camera/action transformations are supplied by wan/utils/cam_utils.py.

Causal Video Backbone

The denoising transformer family lives in wan/modules/model.py, wan/modules/model_causal.py, and wan/modules/model_fast.py. The causal variants preserve temporal state/KV cache across chunks, enabling an unbounded interaction horizon; the fast variant is the released distilled few-step model. Attention implementation and sequence behavior are concentrated in wan/modules/attention.py.

Conditioning and Latent Video Representation

Text prompts are tokenized/encoded through wan/modules/tokenizers.py and wan/modules/t5.py. The input image and generated latent video are encoded/decoded by wan/modules/vae2_1.py. This establishes the core conditioning path: prompt + initial image + interaction/camera trajectories → latent denoiser → decoded frames.

Sampling and Interaction Inputs

Diffusion/flow sampling is implemented in wan/utils/fm_solvers.py and wan/utils/fm_solvers_unipc.py. Action datasets are filesystem-based example bundles such as examples/03/action.npy, examples/03/poses.npy, and examples/03/intrinsics.npy; some scenarios additionally provide keyboard-specific action arrays and prompts. These controls are passed via --action_path, forming an explicit boundary between externally authored trajectories and learned video synthesis.

Distributed Runtime and Dependencies

The intended runtime is PyTorch 2.4+ with FlashAttention, launched via torchrun on multi-GPU hardware. FSDP wrapping and distributed mechanics are in wan/distributed/fsdp.py and wan/distributed/util.py; Ulysses/sequence parallelism is implemented in wan/distributed/ulysses.py and wan/distributed/sequence_parallel.py. Checkpoint weights are external downloads from Hugging Face or ModelScope; deployment is explicitly delegated to external SGLang or NVIDIA flashdreams integrations.