How meituan-longcat/longcat-video works
1. What this repo is
- This is an inference/demo repository for two closely related video generation systems:
- LongCat-Video: text-to-video, image-to-video, video-continuation, and long-video generation.
- LongCat-Video-Avatar: audio-driven character animation with single- and multi-speaker variants.
- The code is organized around model definition, sampling, preprocessing, and runnable demos, not around training or dataset pipelines.
2. Top-level execution surface
- The main user entry points are the run_demo_*.py scripts and run_streamlit.py.
- Each demo script corresponds to a task boundary:
- run_demo_text_to_video.py
- run_demo_image_to_video.py
- run_demo_video_continuation.py
- run_demo_long_video.py
- run_demo_interactive_video.py
- run_demo_avatar_single_audio_to_video.py
- run_demo_avatar_multi_audio_to_video.py
- These scripts load checkpoints, prepare inputs, configure distributed execution, and invoke the appropriate pipeline.
3. Core pipeline boundary
- The main orchestration lives in:
- longcat_video/pipeline_longcat_video.py
- longcat_video/pipeline_longcat_video_avatar.py
- These pipelines sit between the demos and the model internals.
- They are responsible for the end-to-end generation flow: input conditioning, latent initialization, iterative denoising/sampling, decoding, and output assembly.
4. Shared video-generation backbone
- The non-avatar model is implemented under longcat_video/modules/.
- The main backbone is modules/longcat_video_dit.py, which is the central DiT-style video transformer.
- Supporting pieces include:
- modules/blocks.py for transformer/residual block structure
- modules/attention.py for attention logic
- modules/rope_3d.py for 3D rotary positional encoding
- modules/scheduling_flow_match_euler_discrete.py for the sampling schedule
- modules/autoencoder_kl_wan.py for latent video encode/decode
- modules/lora_utils.py for low-rank adaptation hooks
5. Avatar-specific model branch
- Avatar generation has its own model subtree under longcat_video/modules/avatar/.
- Key files:
- longcat_video_dit_avatar.py: avatar-specific DiT backbone
- avatar/attention.py, avatar/blocks.py, avatar/rope_3d.py: variants adapted for the avatar task
- This branch adds audio conditioning and supports both:
- single-audio inputs
- multi-audio inputs
- The avatar pipeline is a superset of the base pipeline, adding audio-driven facial/lip-motion control and character animation constraints.
6. Audio processing subsystem
- Audio conditioning is isolated in longcat_video/audio_process/.
- Notable components:
- wav2vec2.py: audio feature extraction
- torch_utils.py: audio-related tensor utilities
- This subsystem converts raw audio into features that can be injected into the avatar model as conditioning signals.
- In the avatar demos, audio features are the primary extra input beyond text/image/video context.
7. Attention and efficiency architecture
- The repo explicitly includes multiple efficiency mechanisms for long and high-resolution video:
- block_sparse_attention/: block-sparse attention implementation and communication helpers
- context_parallel/: context-parallel utilities for multi-GPU inference
- These are important architectural boundaries because they let the model scale to longer sequences and larger spatial resolutions.
- The README indicates FlashAttention is enabled by default, with support for alternate attention backends via configuration.
8. Distributed inference model
- Multi-GPU execution is driven externally through torchrun, with --context_parallel_size used by the demos.
- The context_parallel package provides the infrastructure needed to split sequence/context work across devices.
- This means the architecture is not just “model + single process,” but a distributed inference stack optimized for long video generation.
9. Input preprocessing and prompting layer
- Repo utilities under longcat_video/utils/ handle task-specific preparation:
- prompt_enhancer.py: prompt augmentation/refinement
- bukcet_config.py (bucket config): resolution/shape bucketing logic for generation
- For avatar generation, the example JSON files under assets/avatar/ define structured inputs combining:
- prompts
- audio file paths
- reference images or video
- task parameters such as segmenting and masking
- This preprocessing layer is the boundary where human-friendly inputs become model-ready conditioning.
10. Latent video generation flow
- The high-level data flow is:
- user prompt / image / audio / optional reference video
- preprocessing and feature extraction
- latent initialization through the autoencoder
- iterative flow-matching / Euler discrete sampling in the DiT backbone
- decode latent outputs back to video frames
- write final video artifacts
- For continuation and long-video modes, the pipeline also manages segmentation and masking across temporal windows.
11. Assets and examples as architectural documentation
- The assets/ directory is not just illustrative; it contains the canonical example inputs used by the demos:
- sample media like motorcycle.mp4
- avatar single/multi audio clips
- example JSON manifests
- These files reveal the expected input contracts for each pipeline and are useful for diagramming the external interface.
12. Runtime and external dependencies that matter
- The project is built around PyTorch and uses GPU-centric inference.
- Material dependencies include:
- flash-attn for optimized attention
- torchaudio / librosa for audio handling
- ffmpeg for media processing
- streamlit for the interactive demo UI
- Hugging Face model downloads for checkpoint distribution
- These tools materially shape the runtime architecture, especially around performance and media I/O.
13. Clear subsystem boundaries
- The repo has a clean separation between:
- demo entry scripts
- task pipelines
- model backbones
- audio preprocessing
- attention/parallelism infrastructure
- utility/config layers
- For an architecture diagram, the main dependency arrows should point from demos into pipelines, from pipelines into model/audio/util subsystems, and then out to decoded media outputs.
14. Overall architectural summary
- Conceptually, this is a modular generative video inference stack with one shared backbone family and two task-oriented front ends:
- a general-purpose video generator
- an audio-conditioned avatar generator
- The dominant design patterns are latent diffusion/flow-based generation, transformer-based denoising, and multi-GPU context-parallel acceleration.
Main entry points: pipeline_longcat_video.py Video pipeline · pipeline_longcat_video_avatar.py Avatar pipeline · longcat_video_dit.py Video DiT · longcat_video_dit_avatar.py Avatar DiT
Components 27
Demo entrypoints
Runnable scripts for each generation task and UI.
- Text-to-video · Entry script for text-conditioned video generation.
run_demo_text_to_video.py - Image-to-video · Entry script for image-conditioned video generation.
run_demo_image_to_video.py - Video continuation · Entry script for extending an existing video clip.
run_demo_video_continuation.py - Long video · Entry script for long-form generation and windowed synthesis.
run_demo_long_video.py - Interactive demo · UI entrypoint for interactive generation workflows.
run_streamlit.py - Avatar single · Single-speaker audio-to-video avatar generation demo.
run_demo_avatar_single_audio_to_video.py - Avatar multi · Multi-speaker audio-to-video avatar generation demo.
run_demo_avatar_multi_audio_to_video.py
Generation pipelines
End-to-end orchestration for non-avatar and avatar inference.
- Video pipeline · Coordinates conditioning, sampling, decoding, and assembly for general video tasks.
longcat_video/pipeline_longcat_video.py - Avatar pipeline · Audio-conditioned pipeline for avatar animation and speaker-aware generation.
longcat_video/pipeline_longcat_video_avatar.py
Shared video backbone
Model, sampling, decoding, and shared transformer components.
- Video DiT · Main diffusion-transformer backbone for non-avatar video generation.
longcat_video/modules/longcat_video_dit.py - Transformer blocks · Residual and transformer block structure shared by the video backbone.
longcat_video/modules/blocks.py - Attention core · Shared attention implementation used by the main model stack.
longcat_video/modules/attention.py - 3D RoPE · Three-dimensional rotary positional encoding for video tokens.
longcat_video/modules/rope_3d.py - Flow schedule · Discrete Euler flow-matching schedule used during sampling.
longcat_video/modules/scheduling_flow_match_euler_discrete.py - Video autoencoder · Encodes and decodes video latents for generation.
longcat_video/modules/autoencoder_kl_wan.py - LoRA hooks · Low-rank adaptation helpers for model injection and fine-tuning style reuse.
longcat_video/modules/lora_utils.py
Avatar branch
Audio-conditioned model variants for character animation.
- Avatar DiT · Avatar-specific transformer backbone with audio-conditioning support.
longcat_video/modules/avatar/longcat_video_dit_avatar.py - Avatar blocks · Avatar-specific block stack for the animation model.
longcat_video/modules/avatar/blocks.py - Avatar attention · Attention variant adapted for avatar generation.
longcat_video/modules/avatar/attention.py - Avatar RoPE · Avatar-specific rotary positional encoding for video tokens.
longcat_video/modules/avatar/rope_3d.py - Audio features · Extracts conditioning features from raw audio for avatar tasks.
longcat_video/audio_process/wav2vec2.py - Audio utils · Tensor and audio helper functions used by the audio conditioning stack.
longcat_video/audio_process/torch_utils.py
Scaling and speed
Distributed and optimized attention infrastructure.
- Block sparse · Block-sparse attention implementation and communication helpers.
longcat_video/block_sparse_attention/bsa_interface.py - Context parallel · Utilities for multi-GPU context/sequence parallel execution.
longcat_video/context_parallel/context_parallel_util.py
Inputs and assets
Prompting, bucketing, and example media/manifests.
- Prompt enhancer · Refines or augments prompts before generation.
longcat_video/utils/prompt_enhancer.py - Bucket config · Resolution and shape bucketing for generation inputs.
longcat_video/utils/bukcet_config.py - Example assets · Canonical sample media, audio clips, and JSON manifests for demos.
assets/avatar
Connections 32
- Text-to-video → Video pipeline: runs · Text-to-video demo invokes the main generation pipeline.
- Image-to-video → Video pipeline: runs · Image-to-video demo invokes the main generation pipeline.
- Video continuation → Video pipeline: runs · Continuation demo invokes the main generation pipeline.
- Long video → Video pipeline: runs · Long-video demo invokes the main generation pipeline.
- Interactive demo → Video pipeline: drives · Streamlit UI routes user actions into the video pipeline.
- Avatar single → Avatar pipeline: runs · Single-audio avatar demo invokes the avatar pipeline.
- Avatar multi → Avatar pipeline: runs · Multi-audio avatar demo invokes the avatar pipeline.
- Video pipeline → Prompt enhancer: preprocesses · Pipelines refine prompts before conditioning the model.
- Video pipeline → Bucket config: buckets · Pipeline selects resolution/shape buckets for inference.
- Video pipeline → Video DiT: denoises · Main pipeline feeds conditioning and latents into the DiT backbone.
- Video pipeline → Video autoencoder: encodes/decodes · Pipeline uses the latent codec around generation.
- Video pipeline → Flow schedule: samples · Pipeline iterates the flow-matching schedule for generation.
- Video pipeline → Context parallel: scales · Pipeline can distribute inference across GPUs.
- Video pipeline → Block sparse: accelerates · Pipeline benefits from block-sparse attention paths.
- Avatar pipeline → Audio features: extracts · Avatar pipeline converts raw audio into conditioning features.
- Avatar pipeline → Audio utils: supports · Avatar pipeline uses audio tensor helpers.
- Avatar pipeline → Avatar DiT: denoises · Avatar pipeline drives the avatar-specific backbone.
- Avatar pipeline → Video autoencoder: encodes/decodes · Avatar pipeline reuses the latent video codec.
- Avatar pipeline → Flow schedule: samples · Avatar pipeline uses the same flow-matching schedule.
- Avatar pipeline → Context parallel: scales · Avatar pipeline supports distributed execution.
- Video DiT → Transformer blocks: built from · The main video transformer is assembled from reusable blocks.
- Video DiT → Attention core: uses · The backbone relies on shared attention internals.
- Video DiT → 3D RoPE: positions · The backbone applies 3D rotary position encoding.
- Video DiT → LoRA hooks: adapts with · The backbone exposes LoRA adaptation hooks.
- Avatar DiT → Avatar blocks: built from · Avatar backbone is assembled from avatar-specific blocks.
- Avatar DiT → Avatar attention: uses · Avatar backbone uses avatar-tuned attention logic.
- Avatar DiT → Avatar RoPE: positions · Avatar backbone applies its own 3D RoPE variant.
- Avatar DiT → Audio features: conditions on · Avatar backbone consumes audio-derived conditioning features.
- Audio features → Audio utils: uses · Audio feature extraction depends on audio tensor utilities.
- Example assets → Avatar single: feeds · Example assets provide canonical inputs for avatar demos.
- Example assets → Avatar multi: feeds · Example assets provide canonical inputs for avatar demos.
- Example assets → Text-to-video: feeds · Sample media and manifests document input expectations.