# meituan-longcat/longcat-video architecture

> Architecture diagram of the GitHub repository meituan-longcat/longcat-video, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/meituan-longcat/longcat-video
- Repository: https://github.com/meituan-longcat/longcat-video

## Overview

1. **What this repo is**
   - This is an **inference/demo repository** for two closely related video generation systems:
     - **LongCat-Video**: text-to-video, image-to-video, video-continuation, and long-video generation.
     - **LongCat-Video-Avatar**: audio-driven character animation with single- and multi-speaker variants.
   - The code is organized around **model definition, sampling, preprocessing, and runnable demos**, not around training or dataset pipelines.

2. **Top-level execution surface**
   - The main user entry points are the `run_demo_*.py` scripts and `run_streamlit.py`.
   - Each demo script corresponds to a task boundary:
     - `run_demo_text_to_video.py`
     - `run_demo_image_to_video.py`
     - `run_demo_video_continuation.py`
     - `run_demo_long_video.py`
     - `run_demo_interactive_video.py`
     - `run_demo_avatar_single_audio_to_video.py`
     - `run_demo_avatar_multi_audio_to_video.py`
   - These scripts load checkpoints, prepare inputs, configure distributed execution, and invoke the appropriate pipeline.

3. **Core pipeline boundary**
   - The main orchestration lives in:
     - `longcat_video/pipeline_longcat_video.py`
     - `longcat_video/pipeline_longcat_video_avatar.py`
   - These pipelines sit between the demos and the model internals.
   - They are responsible for the end-to-end generation flow: input conditioning, latent initialization, iterative denoising/sampling, decoding, and output assembly.

4. **Shared video-generation backbone**
   - The non-avatar model is implemented under `longcat_video/modules/`.
   - The main backbone is `modules/longcat_video_dit.py`, which is the central **DiT-style video transformer**.
   - Supporting pieces include:
     - `modules/blocks.py` for transformer/residual block structure
     - `modules/attention.py` for attention logic
     - `modules/rope_3d.py` for 3D rotary positional encoding
     - `modules/scheduling_flow_match_euler_discrete.py` for the sampling schedule
     - `modules/autoencoder_kl_wan.py` for latent video encode/decode
     - `modules/lora_utils.py` for low-rank adaptation hooks

5. **Avatar-specific model branch**
   - Avatar generation has its own model subtree under `longcat_video/modules/avatar/`.
   - Key files:
     - `longcat_video_dit_avatar.py`: avatar-specific DiT backbone
     - `avatar/attention.py`, `avatar/blocks.py`, `avatar/rope_3d.py`: variants adapted for the avatar task
   - This branch adds audio conditioning and supports both:
     - **single-audio** inputs
     - **multi-audio** inputs
   - The avatar pipeline is a superset of the base pipeline, adding audio-driven facial/lip-motion control and character animation constraints.

6. **Audio processing subsystem**
   - Audio conditioning is isolated in `longcat_video/audio_process/`.
   - Notable components:
     - `wav2vec2.py`: audio feature extraction
     - `torch_utils.py`: audio-related tensor utilities
   - This subsystem converts raw audio into features that can be injected into the avatar model as conditioning signals.
   - In the avatar demos, audio features are the primary extra input beyond text/image/video context.

7. **Attention and efficiency architecture**
   - The repo explicitly includes multiple efficiency mechanisms for long and high-resolution video:
     - `block_sparse_attention/`: block-sparse attention implementation and communication helpers
     - `context_parallel/`: context-parallel utilities for multi-GPU inference
   - These are important architectural boundaries because they let the model scale to longer sequences and larger spatial resolutions.
   - The README indicates FlashAttention is enabled by default, with support for alternate attention backends via configuration.

8. **Distributed inference model**
   - Multi-GPU execution is driven externally through `torchrun`, with `--context_parallel_size` used by the demos.
   - The `context_parallel` package provides the infrastructure needed to split sequence/context work across devices.
   - This means the architecture is not just “model + single process,” but a **distributed inference stack** optimized for long video generation.

9. **Input preprocessing and prompting layer**
   - Repo utilities under `longcat_video/utils/` handle task-specific preparation:
     - `prompt_enhancer.py`: prompt augmentation/refinement
     - `bukcet_config.py` (bucket config): resolution/shape bucketing logic for generation
   - For avatar generation, the example JSON files under `assets/avatar/` define structured inputs combining:
     - prompts
     - audio file paths
     - reference images or video
     - task parameters such as segmenting and masking
   - This preprocessing layer is the boundary where human-friendly inputs become model-ready conditioning.

10. **Latent video generation flow**
   - The high-level data flow is:
     - user prompt / image / audio / optional reference video
     - preprocessing and feature extraction
     - latent initialization through the autoencoder
     - iterative flow-matching / Euler discrete sampling in the DiT backbone
     - decode latent outputs back to video frames
     - write final video artifacts
   - For continuation and long-video modes, the pipeline also manages segmentation and masking across temporal windows.

11. **Assets and examples as architectural documentation**
   - The `assets/` directory is not just illustrative; it contains the canonical example inputs used by the demos:
     - sample media like `motorcycle.mp4`
     - avatar single/multi audio clips
     - example JSON manifests
   - These files reveal the expected input contracts for each pipeline and are useful for diagramming the external interface.

12. **Runtime and external dependencies that matter**
   - The project is built around **PyTorch** and uses GPU-centric inference.
   - Material dependencies include:
     - `flash-attn` for optimized attention
     - `torchaudio` / `librosa` for audio handling
     - `ffmpeg` for media processing
     - `streamlit` for the interactive demo UI
     - Hugging Face model downloads for checkpoint distribution
   - These tools materially shape the runtime architecture, especially around performance and media I/O.

13. **Clear subsystem boundaries**
   - The repo has a clean separation between:
     - **demo entry scripts**
     - **task pipelines**
     - **model backbones**
     - **audio preprocessing**
     - **attention/parallelism infrastructure**
     - **utility/config layers**
   - For an architecture diagram, the main dependency arrows should point from demos into pipelines, from pipelines into model/audio/util subsystems, and then out to decoded media outputs.

14. **Overall architectural summary**
   - Conceptually, this is a **modular generative video inference stack** with one shared backbone family and two task-oriented front ends:
     - a general-purpose video generator
     - an audio-conditioned avatar generator
   - The dominant design patterns are **latent diffusion/flow-based generation, transformer-based denoising, and multi-GPU context-parallel acceleration**.

Main entry points: [`longcat_video/pipeline_longcat_video.py`](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/pipeline_longcat_video.py) (Video pipeline), [`longcat_video/pipeline_longcat_video_avatar.py`](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/pipeline_longcat_video_avatar.py) (Avatar pipeline), [`longcat_video/modules/longcat_video_dit.py`](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/longcat_video_dit.py) (Video DiT), [`longcat_video/modules/avatar/longcat_video_dit_avatar.py`](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/avatar/longcat_video_dit_avatar.py) (Avatar DiT)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_demos["Demo entrypoints"]
  node_node_text_demo["Text-to-video<br/>demo script"]
  node_node_image_demo["Image-to-video<br/>demo script"]
  node_node_cont_demo["Video continuation<br/>demo script"]
  node_node_long_demo["Long video<br/>demo script"]
  node_node_interactive_demo["Interactive demo<br/>streamlit app<br/>[run_streamlit.py]"]
  node_node_avatar_single_demo["Avatar single<br/>demo script"]
  node_node_avatar_multi_demo["Avatar multi<br/>demo script"]
end

subgraph group_grp_pipelines["Generation pipelines"]
  node_node_video_pipeline["Video pipeline<br/>generation pipeline"]
  node_node_avatar_pipeline["Avatar pipeline<br/>generation pipeline"]
end

subgraph group_grp_core["Shared video backbone"]
  node_node_dit["Video DiT<br/>transformer backbone"]
  node_node_blocks["Transformer blocks<br/>model building blocks<br/>[blocks.py]"]
  node_node_attention["Attention core<br/>attention module<br/>[attention.py]"]
  node_node_rope["3D RoPE<br/>positional encoding<br/>[rope_3d.py]"]
  node_node_scheduler["Flow schedule<br/>sampler schedule"]
  node_node_autoencoder[("Video autoencoder<br/>latent codec")]
  node_node_lora["LoRA hooks<br/>adaptation utils<br/>[lora_utils.py]"]
end

subgraph group_grp_avatar["Avatar branch"]
  node_node_avatar_dit["Avatar DiT<br/>avatar backbone"]
  node_node_avatar_blocks["Avatar blocks<br/>avatar model blocks<br/>[blocks.py]"]
  node_node_avatar_attention["Avatar attention<br/>[attention.py]"]
  node_node_avatar_rope["Avatar RoPE<br/>avatar positional encoding<br/>[rope_3d.py]"]
  node_node_audio["Audio features<br/>audio preprocessing<br/>[wav2vec2.py]"]
  node_node_audio_utils["Audio utils<br/>audio utilities<br/>[torch_utils.py]"]
end

subgraph group_grp_speed["Scaling and speed"]
  node_node_block_sparse["Block sparse<br/>attention acceleration<br/>[bsa_interface.py]"]
  node_node_context_parallel["Context parallel<br/>distributed inference"]
end

subgraph group_grp_io["Inputs and assets"]
  node_node_prompt["Prompt enhancer<br/>input utility<br/>[prompt_enhancer.py]"]
  node_node_bucket["Bucket config<br/>shape config<br/>[bukcet_config.py]"]
  node_node_assets["Example assets<br/>media and manifests"]
end

node_node_text_demo -->|"runs"| node_node_video_pipeline
node_node_image_demo -->|"runs"| node_node_video_pipeline
node_node_cont_demo -->|"runs"| node_node_video_pipeline
node_node_long_demo -->|"runs"| node_node_video_pipeline
node_node_interactive_demo -->|"drives"| node_node_video_pipeline
node_node_avatar_single_demo -->|"runs"| node_node_avatar_pipeline
node_node_avatar_multi_demo -->|"runs"| node_node_avatar_pipeline
node_node_video_pipeline -->|"preprocesses"| node_node_prompt
node_node_video_pipeline -->|"buckets"| node_node_bucket
node_node_video_pipeline -->|"denoises"| node_node_dit
node_node_video_pipeline -->|"encodes/decodes"| node_node_autoencoder
node_node_video_pipeline -->|"samples"| node_node_scheduler
node_node_video_pipeline -->|"scales"| node_node_context_parallel
node_node_video_pipeline -->|"accelerates"| node_node_block_sparse
node_node_avatar_pipeline -->|"extracts"| node_node_audio
node_node_avatar_pipeline -->|"supports"| node_node_audio_utils
node_node_avatar_pipeline -->|"denoises"| node_node_avatar_dit
node_node_avatar_pipeline -->|"encodes/decodes"| node_node_autoencoder
node_node_avatar_pipeline -->|"samples"| node_node_scheduler
node_node_avatar_pipeline -->|"scales"| node_node_context_parallel
node_node_dit -->|"built from"| node_node_blocks
node_node_dit -->|"uses"| node_node_attention
node_node_dit -->|"positions"| node_node_rope
node_node_dit -->|"adapts with"| node_node_lora
node_node_avatar_dit -->|"built from"| node_node_avatar_blocks
node_node_avatar_dit -->|"uses"| node_node_avatar_attention
node_node_avatar_dit -->|"positions"| node_node_avatar_rope
node_node_avatar_dit -->|"conditions on"| node_node_audio
node_node_audio -->|"uses"| node_node_audio_utils
node_node_assets -->|"feeds"| node_node_avatar_single_demo
node_node_assets -->|"feeds"| node_node_avatar_multi_demo
node_node_assets -->|"feeds"| node_node_text_demo

click node_node_text_demo "https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_text_to_video.py"
click node_node_image_demo "https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_image_to_video.py"
click node_node_cont_demo "https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_video_continuation.py"
click node_node_long_demo "https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_long_video.py"
click node_node_interactive_demo "https://github.com/meituan-longcat/longcat-video/blob/main/run_streamlit.py"
click node_node_avatar_single_demo "https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_avatar_single_audio_to_video.py"
click node_node_avatar_multi_demo "https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_avatar_multi_audio_to_video.py"
click node_node_video_pipeline "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/pipeline_longcat_video.py"
click node_node_avatar_pipeline "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/pipeline_longcat_video_avatar.py"
click node_node_dit "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/longcat_video_dit.py"
click node_node_blocks "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/blocks.py"
click node_node_attention "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/attention.py"
click node_node_rope "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/rope_3d.py"
click node_node_scheduler "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/scheduling_flow_match_euler_discrete.py"
click node_node_autoencoder "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/autoencoder_kl_wan.py"
click node_node_lora "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/lora_utils.py"
click node_node_avatar_dit "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/avatar/longcat_video_dit_avatar.py"
click node_node_avatar_blocks "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/avatar/blocks.py"
click node_node_avatar_attention "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/avatar/attention.py"
click node_node_avatar_rope "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/avatar/rope_3d.py"
click node_node_audio "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/audio_process/wav2vec2.py"
click node_node_audio_utils "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/audio_process/torch_utils.py"
click node_node_block_sparse "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/block_sparse_attention/bsa_interface.py"
click node_node_context_parallel "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/context_parallel/context_parallel_util.py"
click node_node_prompt "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/utils/prompt_enhancer.py"
click node_node_bucket "https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/utils/bukcet_config.py"
click node_node_assets "https://github.com/meituan-longcat/longcat-video/tree/main/assets/avatar"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_text_demo,node_node_image_demo,node_node_cont_demo,node_node_long_demo,node_node_interactive_demo,node_node_avatar_single_demo,node_node_avatar_multi_demo toneBlue
class node_node_video_pipeline,node_node_avatar_pipeline toneAmber
class node_node_dit,node_node_blocks,node_node_attention,node_node_rope,node_node_scheduler,node_node_autoencoder,node_node_lora toneMint
class node_node_avatar_dit,node_node_avatar_blocks,node_node_avatar_attention,node_node_avatar_rope,node_node_audio,node_node_audio_utils toneRose
class node_node_block_sparse,node_node_context_parallel toneIndigo
class node_node_prompt,node_node_bucket,node_node_assets toneTeal
```

## Components

### Demo entrypoints

Runnable scripts for each generation task and UI.

- [Text-to-video](https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_text_to_video.py): demo script · Entry script for text-conditioned video generation. · `run_demo_text_to_video.py`
- [Image-to-video](https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_image_to_video.py): demo script · Entry script for image-conditioned video generation. · `run_demo_image_to_video.py`
- [Video continuation](https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_video_continuation.py): demo script · Entry script for extending an existing video clip. · `run_demo_video_continuation.py`
- [Long video](https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_long_video.py): demo script · Entry script for long-form generation and windowed synthesis. · `run_demo_long_video.py`
- [Interactive demo](https://github.com/meituan-longcat/longcat-video/blob/main/run_streamlit.py): streamlit app · UI entrypoint for interactive generation workflows. · `run_streamlit.py`
- [Avatar single](https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_avatar_single_audio_to_video.py): demo script · Single-speaker audio-to-video avatar generation demo. · `run_demo_avatar_single_audio_to_video.py`
- [Avatar multi](https://github.com/meituan-longcat/longcat-video/blob/main/run_demo_avatar_multi_audio_to_video.py): demo script · Multi-speaker audio-to-video avatar generation demo. · `run_demo_avatar_multi_audio_to_video.py`

### Generation pipelines

End-to-end orchestration for non-avatar and avatar inference.

- [Video pipeline](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/pipeline_longcat_video.py): generation pipeline · Coordinates conditioning, sampling, decoding, and assembly for general video tasks. · `longcat_video/pipeline_longcat_video.py`
- [Avatar pipeline](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/pipeline_longcat_video_avatar.py): generation pipeline · Audio-conditioned pipeline for avatar animation and speaker-aware generation. · `longcat_video/pipeline_longcat_video_avatar.py`

### Shared video backbone

Model, sampling, decoding, and shared transformer components.

- [Video DiT](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/longcat_video_dit.py): transformer backbone · Main diffusion-transformer backbone for non-avatar video generation. · `longcat_video/modules/longcat_video_dit.py`
- [Transformer blocks](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/blocks.py): model building blocks · Residual and transformer block structure shared by the video backbone. · `longcat_video/modules/blocks.py`
- [Attention core](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/attention.py): attention module · Shared attention implementation used by the main model stack. · `longcat_video/modules/attention.py`
- [3D RoPE](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/rope_3d.py): positional encoding · Three-dimensional rotary positional encoding for video tokens. · `longcat_video/modules/rope_3d.py`
- [Flow schedule](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/scheduling_flow_match_euler_discrete.py): sampler schedule · Discrete Euler flow-matching schedule used during sampling. · `longcat_video/modules/scheduling_flow_match_euler_discrete.py`
- [Video autoencoder](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/autoencoder_kl_wan.py): latent codec · Encodes and decodes video latents for generation. · `longcat_video/modules/autoencoder_kl_wan.py`
- [LoRA hooks](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/lora_utils.py): adaptation utils · Low-rank adaptation helpers for model injection and fine-tuning style reuse. · `longcat_video/modules/lora_utils.py`

### Avatar branch

Audio-conditioned model variants for character animation.

- [Avatar DiT](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/avatar/longcat_video_dit_avatar.py): avatar backbone · Avatar-specific transformer backbone with audio-conditioning support. · `longcat_video/modules/avatar/longcat_video_dit_avatar.py`
- [Avatar blocks](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/avatar/blocks.py): avatar model blocks · Avatar-specific block stack for the animation model. · `longcat_video/modules/avatar/blocks.py`
- [Avatar attention](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/avatar/attention.py): avatar attention · Attention variant adapted for avatar generation. · `longcat_video/modules/avatar/attention.py`
- [Avatar RoPE](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/modules/avatar/rope_3d.py): avatar positional encoding · Avatar-specific rotary positional encoding for video tokens. · `longcat_video/modules/avatar/rope_3d.py`
- [Audio features](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/audio_process/wav2vec2.py): audio preprocessing · Extracts conditioning features from raw audio for avatar tasks. · `longcat_video/audio_process/wav2vec2.py`
- [Audio utils](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/audio_process/torch_utils.py): audio utilities · Tensor and audio helper functions used by the audio conditioning stack. · `longcat_video/audio_process/torch_utils.py`

### Scaling and speed

Distributed and optimized attention infrastructure.

- [Block sparse](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/block_sparse_attention/bsa_interface.py): attention acceleration · Block-sparse attention implementation and communication helpers. · `longcat_video/block_sparse_attention/bsa_interface.py`
- [Context parallel](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/context_parallel/context_parallel_util.py): distributed inference · Utilities for multi-GPU context/sequence parallel execution. · `longcat_video/context_parallel/context_parallel_util.py`

### Inputs and assets

Prompting, bucketing, and example media/manifests.

- [Prompt enhancer](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/utils/prompt_enhancer.py): input utility · Refines or augments prompts before generation. · `longcat_video/utils/prompt_enhancer.py`
- [Bucket config](https://github.com/meituan-longcat/longcat-video/blob/main/longcat_video/utils/bukcet_config.py): shape config · Resolution and shape bucketing for generation inputs. · `longcat_video/utils/bukcet_config.py`
- [Example assets](https://github.com/meituan-longcat/longcat-video/tree/main/assets/avatar): media and manifests · Canonical sample media, audio clips, and JSON manifests for demos. · `assets/avatar`

## Connections

- Text-to-video → Video pipeline: runs · Text-to-video demo invokes the main generation pipeline.
- Image-to-video → Video pipeline: runs · Image-to-video demo invokes the main generation pipeline.
- Video continuation → Video pipeline: runs · Continuation demo invokes the main generation pipeline.
- Long video → Video pipeline: runs · Long-video demo invokes the main generation pipeline.
- Interactive demo → Video pipeline: drives · Streamlit UI routes user actions into the video pipeline.
- Avatar single → Avatar pipeline: runs · Single-audio avatar demo invokes the avatar pipeline.
- Avatar multi → Avatar pipeline: runs · Multi-audio avatar demo invokes the avatar pipeline.
- Video pipeline → Prompt enhancer: preprocesses · Pipelines refine prompts before conditioning the model.
- Video pipeline → Bucket config: buckets · Pipeline selects resolution/shape buckets for inference.
- Video pipeline → Video DiT: denoises · Main pipeline feeds conditioning and latents into the DiT backbone.
- Video pipeline → Video autoencoder: encodes/decodes · Pipeline uses the latent codec around generation.
- Video pipeline → Flow schedule: samples · Pipeline iterates the flow-matching schedule for generation.
- Video pipeline → Context parallel: scales · Pipeline can distribute inference across GPUs.
- Video pipeline → Block sparse: accelerates · Pipeline benefits from block-sparse attention paths.
- Avatar pipeline → Audio features: extracts · Avatar pipeline converts raw audio into conditioning features.
- Avatar pipeline → Audio utils: supports · Avatar pipeline uses audio tensor helpers.
- Avatar pipeline → Avatar DiT: denoises · Avatar pipeline drives the avatar-specific backbone.
- Avatar pipeline → Video autoencoder: encodes/decodes · Avatar pipeline reuses the latent video codec.
- Avatar pipeline → Flow schedule: samples · Avatar pipeline uses the same flow-matching schedule.
- Avatar pipeline → Context parallel: scales · Avatar pipeline supports distributed execution.
- Video DiT → Transformer blocks: built from · The main video transformer is assembled from reusable blocks.
- Video DiT → Attention core: uses · The backbone relies on shared attention internals.
- Video DiT → 3D RoPE: positions · The backbone applies 3D rotary position encoding.
- Video DiT → LoRA hooks: adapts with · The backbone exposes LoRA adaptation hooks.
- Avatar DiT → Avatar blocks: built from · Avatar backbone is assembled from avatar-specific blocks.
- Avatar DiT → Avatar attention: uses · Avatar backbone uses avatar-tuned attention logic.
- Avatar DiT → Avatar RoPE: positions · Avatar backbone applies its own 3D RoPE variant.
- Avatar DiT → Audio features: conditions on · Avatar backbone consumes audio-derived conditioning features.
- Audio features → Audio utils: uses · Audio feature extraction depends on audio tensor utilities.
- Example assets → Avatar single: feeds · Example assets provide canonical inputs for avatar demos.
- Example assets → Avatar multi: feeds · Example assets provide canonical inputs for avatar demos.
- Example assets → Text-to-video: feeds · Sample media and manifests document input expectations.
