# skyworkai/skyreels-v2 architecture

> Architecture diagram of the GitHub repository skyworkai/skyreels-v2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skyworkai/skyreels-v2
- Repository: https://github.com/skyworkai/skyreels-v2

## Overview

##### Project Type and Purpose
- This repository is an open-source video generative inference framework (SkyReels-V2) plus a companion video-captioning tool (SkyCaptioner-V1).
- Its core purpose is to run infinite-length video generation (text-to-video, image-to-video, diffusion forcing) and to perform video captioning on input video clips.

##### High-Level File Structure Analysis
- Top Level
– generate_video_df.py / generate_video.py: CLI entry points for video generation (diffusion forcing vs. standard pipelines).
– requirements.txt: global Python dependencies.
– skycaptioner_v1/: standalone video captioner tool with its own examples, scripts, and requirements.
– skyreels_v2_infer/: core inference library for SkyReels-V2, containing:
- distributed/: xDiT USP multi-GPU parallelization support.
- modules/: model building blocks (Attention, Transformer, VAE, CLIP, T5, XLM-RoBERTa, Tokenizers).
- pipelines/: high-level inference pipelines (diffusion_forcing, text2video, image2video, prompt_enhancer).
- scheduler/: flow-matching/solvers for diffusion forcing.

##### Key Architectural Patterns
- Modular pipes + plug-in schedulers: each pipeline class orchestrates components from modules/ and scheduler/.
- CLI → Pipeline → Modules → Scheduler → VAE decoder pattern.
- Optional distributed/Multi-GPU via xDiT USP.
- Optional prompt enhancer step using an external LLM (Qwen2.5-32B).
- Separation between generation and captioning subsystems.

##### Main Components & Interactions (
a. User Interface Layer
- CLI scripts (generate_video_df.py, generate_video.py, infer_*.sh)
- Input: prompt, optional image, model_id, inference params.

b. Pipeline Layer (skyreels_v2_infer/pipelines)
- Prompt Enhancer (optional)
- Text2VideoPipeline
- Image2VideoPipeline
- DiffusionForcingPipeline

c. Core Model Layer (skyreels_v2_infer/modules)
- Tokenization (tokenizers.py)
- Text encoders (T5, XLM-RoBERTa)
- Visual encoders (CLIP)
- Diffusion Transformer (transformer.py + attention.py)
- VAE (vae.py)

d. Scheduler Layer (skyreels_v2_infer/scheduler)
- Flow Matching / Diffusion Forcing solvers

e. Distributed Layer (skyreels_v2_infer/distributed)
- xDiT USP parallelization component

f. Video Captioning Subsystem (skycaptioner_v1)
- Video Captioner model wrappers (vllm_fusion_caption, vllm_struct_caption)
- Utility scripts (utils.py)
- Example data and inference shells

g. External Dependencies
- Hugging Face Model Hub (download model weights)
- Offload to CPU/Accelerator
- Python libraries (torch, diffusers, vllm, etc.)

##### Data Flow & Dependencies
1. User runs CLI →
2. (Optional) Prompt Enhancer pipeline enriches text →
3. Tokenizers / Text Encoder convert prompt →
4. (For I2V) CLIP encodes input image →
5. Pipeline assembles condition embeddings and latent frames →
6. Scheduler drives denoising loops →
7. Diffusion Transformer updates latent video tokens →
8. VAE decodes latent tokens into pixel frames →
9. Frames are assembled into output video file.
10. (Captioning) Video file input → SkyCaptioner scripts → text captions output.

##### Architectural Patterns & Principles
- Single-responsibility modules
- Pipeline orchestration for reusability
- Clear separation between inference stages (conditioning, diffusion, decoding)
- Plug-and-play scheduler strategies (synchronous vs. async diffusion forcing)
- Configurable offload and multi-GPU support

##### Essential Elements to Capture
- CLI entry point and parameter inputs
- Pipeline orchestration layer and its sub-pipelines
- Core model blocks (tokenizers, encoders, transformer, VAE)
- Scheduler driving the diffusion process
- Multi-GPU/distributed integration via xDiT USP
- Separate captioning tool subtree

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface Layer
    subgraph "User Interface"
        GEN_STD["generate_video.py"]:::ui
        GEN_DF["generate_video_df.py"]:::ui
    end

    %% Pipeline Layer
    subgraph "Pipeline Layer"
        PE["Prompt Enhancer"]:::pipeline
        T2V["Text2VideoPipeline"]:::pipeline
        I2V["Image2VideoPipeline"]:::pipeline
        DF["DiffusionForcingPipeline"]:::pipeline
    end

    %% Core Model Layer
    subgraph "Core Model Modules"
        TK["Tokenization Module"]:::modules
        T5["T5 Text Encoder"]:::modules
        XLM["XLM-RoBERTa Text Encoder"]:::modules
        CLIP["CLIP Visual Encoder"]:::modules
        ATT["Attention Mechanism"]:::modules
        TRF["Diffusion Transformer"]:::modules
        VAE["VAE Decoder"]:::modules
    end

    %% Scheduler Layer
    subgraph "Scheduler Layer"
        SCH["Flow Matching / Diffusion Forcing Solvers"]:::scheduler
    end

    %% Distributed Support
    subgraph "Distributed Layer"
        DIST["xDiT USP Parallelization"]:::dist
    end

    %% Video Captioning Subsystem
    subgraph "Video Captioning Subsystem"
        VCF["vllm_fusion_caption.py"]:::caption
        ICF["infer_fusion_caption.sh"]:::caption
        VCS["vllm_struct_caption.py"]:::caption
        ICS["infer_struct_caption.sh"]:::caption
        UT["utils.py"]:::caption
    end

    %% External Dependencies
    HF["Hugging Face Model Hub"]:::external
    OFF["Offload to CPU/GPU"]:::external
    TR["Torchrun (Multi-GPU)"]:::external

    %% Data Flow
    GEN_STD -->|runs CLI| PE
    GEN_DF -->|runs CLI| PE
    PE -->|enhanced prompt| T2V
    PE -->|enhanced prompt| I2V
    PE -->|enhanced prompt| DF
    T2V -->|text embeddings| TK
    I2V -->|image features| CLIP
    DF -->|conditioning| TK
    TK --> T5
    TK --> XLM
    CLIP --> TRF
    T5 --> TRF
    XLM --> TRF
    TRF --> SCH
    SCH --> VAE
    VAE -->|decoded frames| OUT["Output Video File"]:::external
    OUT -->|for captioning| VCF
    VCF --> ICF
    OUT --> VCS
    VCS --> ICS
    UT --> VCF
    UT --> VCS

    %% Distributed and External Links
    GEN_STD --> TR
    GEN_DF --> TR
    TR --> DIST
    DIST --> SCH
    SCH --> OFF
    HF --> TK
    HF --> CLIP
    HF --> T5
    HF --> XLM
    HF --> VAE

    %% Click events
    click GEN_STD "https://github.com/skyworkai/skyreels-v2/blob/main/generate_video.py"
    click GEN_DF "https://github.com/skyworkai/skyreels-v2/blob/main/generate_video_df.py"
    click PE "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/pipelines/prompt_enhancer.py"
    click T2V "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/pipelines/text2video_pipeline.py"
    click I2V "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/pipelines/image2video_pipeline.py"
    click DF "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/pipelines/diffusion_forcing_pipeline.py"
    click TK "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/modules/tokenizers.py"
    click T5 "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/modules/t5.py"
    click XLM "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/modules/xlm_roberta.py"
    click CLIP "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/modules/clip.py"
    click ATT "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/modules/attention.py"
    click TRF "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/modules/transformer.py"
    click VAE "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/modules/vae.py"
    click SCH "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/scheduler/fm_solvers_unipc.py"
    click DIST "https://github.com/skyworkai/skyreels-v2/blob/main/skyreels_v2_infer/distributed/xdit_context_parallel.py"
    click VCF "https://github.com/skyworkai/skyreels-v2/blob/main/skycaptioner_v1/scripts/vllm_fusion_caption.py"
    click ICF "https://github.com/skyworkai/skyreels-v2/blob/main/skycaptioner_v1/infer_fusion_caption.sh"
    click VCS "https://github.com/skyworkai/skyreels-v2/blob/main/skycaptioner_v1/scripts/vllm_struct_caption.py"
    click ICS "https://github.com/skyworkai/skyreels-v2/blob/main/skycaptioner_v1/infer_struct_caption.sh"
    click UT "https://github.com/skyworkai/skyreels-v2/blob/main/skycaptioner_v1/scripts/utils.py"

    %% Styles
    classDef ui fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef pipeline fill:#90EE90,stroke:#333,stroke-width:1px
    classDef modules fill:#FFA500,stroke:#333,stroke-width:1px
    classDef scheduler fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef dist fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef caption fill:#FFFFE0,stroke:#333,stroke-width:1px
    classDef external fill:#EEE,stroke:#333,stroke-width:1px
```
