How skyworkai/skyreels-v2 works
Project Type and Purpose
• This repository is an open-source video generative inference framework (SkyReels-V2) plus a companion video-captioning tool (SkyCaptioner-V1).
• Its core purpose is to run infinite-length video generation (text-to-video, image-to-video, diffusion forcing) and to perform video captioning on input video clips.
High-Level File Structure Analysis
• Top Level
– generate_video_df.py / generate_video.py: CLI entry points for video generation (diffusion forcing vs. standard pipelines).
– requirements.txt: global Python dependencies.
– skycaptioner_v1/: standalone video captioner tool with its own examples, scripts, and requirements.
– skyreels_v2_infer/: core inference library for SkyReels-V2, containing:
• distributed/: xDiT USP multi-GPU parallelization support.
• modules/: model building blocks (Attention, Transformer, VAE, CLIP, T5, XLM-RoBERTa, Tokenizers).
• pipelines/: high-level inference pipelines (diffusion_forcing, text2video, image2video, prompt_enhancer).
• scheduler/: flow-matching/solvers for diffusion forcing.
Key Architectural Patterns
• Modular pipes + plug-in schedulers: each pipeline class orchestrates components from modules/ and scheduler/.
• CLI → Pipeline → Modules → Scheduler → VAE decoder pattern.
• Optional distributed/Multi-GPU via xDiT USP.
• Optional prompt enhancer step using an external LLM (Qwen2.5-32B).
• Separation between generation and captioning subsystems.
Main Components & Interactions (
a. User Interface Layer
• CLI scripts (generate_video_df.py, generate_video.py, infer_*.sh)
• Input: prompt, optional image, model_id, inference params.
b. Pipeline Layer (skyreels_v2_infer/pipelines)
• Prompt Enhancer (optional)
• Text2VideoPipeline
• Image2VideoPipeline
• DiffusionForcingPipeline
c. Core Model Layer (skyreels_v2_infer/modules)
• Tokenization (tokenizers.py)
• Text encoders (T5, XLM-RoBERTa)
• Visual encoders (CLIP)
• Diffusion Transformer (transformer.py + attention.py)
• VAE (vae.py)
d. Scheduler Layer (skyreels_v2_infer/scheduler)
• Flow Matching / Diffusion Forcing solvers
e. Distributed Layer (skyreels_v2_infer/distributed)
• xDiT USP parallelization component
f. Video Captioning Subsystem (skycaptioner_v1)
• Video Captioner model wrappers (vllm_fusion_caption, vllm_struct_caption)
• Utility scripts (utils.py)
• Example data and inference shells
g. External Dependencies
• Hugging Face Model Hub (download model weights)
• Offload to CPU/Accelerator
• Python libraries (torch, diffusers, vllm, etc.)
Data Flow & Dependencies
1. User runs CLI →
2. (Optional) Prompt Enhancer pipeline enriches text →
3. Tokenizers / Text Encoder convert prompt →
4. (For I2V) CLIP encodes input image →
5. Pipeline assembles condition embeddings and latent frames →
6. Scheduler drives denoising loops →
7. Diffusion Transformer updates latent video tokens →
8. VAE decodes latent tokens into pixel frames →
9. Frames are assembled into output video file.
10. (Captioning) Video file input → SkyCaptioner scripts → text captions output.
Architectural Patterns & Principles
• Single-responsibility modules
• Pipeline orchestration for reusability
• Clear separation between inference stages (conditioning, diffusion, decoding)
• Plug-and-play scheduler strategies (synchronous vs. async diffusion forcing)
• Configurable offload and multi-GPU support
Essential Elements to Capture
• CLI entry point and parameter inputs
• Pipeline orchestration layer and its sub-pipelines
• Core model blocks (tokenizers, encoders, transformer, VAE)
• Scheduler driving the diffusion process
• Multi-GPU/distributed integration via xDiT USP
• Separate captioning tool subtree