# liuff19/video-t1 architecture

> Architecture diagram of the GitHub repository liuff19/video-t1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/liuff19/video-t1
- Repository: https://github.com/liuff19/video-t1

## Overview

##### Main Components
- Input/Interface Layer: This is where the user provides a text prompt. In the code, this is most clearly shown in files like videot1.py, videot1_multigpu.py, and videot1_woimg.py.
- Video Generation Pipeline:
- Pipeline Module: The pipeline folder (e.g., pipeline/videot1_pipeline.py) sets up the end-to-end video generation process.
- Pyramid-Flow Engine: The pyramid_flow directory is the core of the video generation engine. It contains several sub-modules:
▪ Diffusion Schedulers: Files in pyramid_flow/diffusion_schedulers govern various denoising strategies (e.g., scheduling_cosine_ddpm.py, scheduling_flow_matching.py).
▪ Pyramid DiT: The pyramid_flow/pyramid_dit module implements Transformer-like models (e.g., flux_modules and mmdit_modules) for embedding, normalization, and the pyramid design.
▪ Video VAE: Located in pyramid_flow/video_vae, these modules implement VAE and discriminator networks which focus on the video generation’s temporal dynamics.
- Reward and Guidance Module: A VisionReward model is used (see references in the README and vqa folder for some related data) to provide feedback. This is coupled with the optional Image-CoT generation (image branch) and hierarchical prompts (supported by additional scripts and checkpoints that need to be loaded).
- Trainer/Utility Modules: The pyramid_flow/trainer_misc contains supportive functionalities for training and model communication (e.g., fsdp_trainer.py, vae_ddp_trainer.py). Also, the utils folder holds helper functions like hierarchical_prompts.
- Scripts and Multi-GPU Support: The pyramid_flow/scripts contains scripts (such as app_multigpu_engine.py) and shell scripts for training and inference workflows; videot1_multigpu.py enables the loading of different models on distinct GPUs.
- External Dependencies and Checkpoints: The README instructs downloading external checkpoints for Pyramid-Flow, VisionReward, Image-CoT, and optionally hierarchical prompt models (DeepSeek-R1-Distill-Llama-8B).

- In the Video Generation Pipeline, represent sub-components:
- Data Preprocessing (if present, implied by initial prompt handling).
- Pyramid-Flow Engine – include separate blocks for:
– Diffusion Schedulers (illustrate the two types: cosine DDPM and flow matching).
– Pyramid DiT blocks split into Flux (for one branch of processing) and MMDiT modules (for an alternative branch or modification).
– Video VAE module for the temporal synthesis part.
- Route the outputs of the Pyramid-Flow Engine into the “Test-Time Scaling” module which refines the result by employing TTS strategies (as described: random linear search and Tree of Frames search with three stages).

##### Architectural Patterns and Key Details
- MVC-like separation: Input → Processing (Pipeline with distinct modules) → Evaluation (Reward feedback) → Inference Output.
- Modular design: Each sub-module (diffusion schedulers, pyramid_dit, video_vae, etc.) encapsulates a specific stage in the generation process.
- Underline the importance of test-time iterative strategies: The TTS approach uses a combination of linear sampling and hierarchical tree strategies for evaluating video generation quality.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input/Interface Layer
    subgraph "Input/Interface Layer"
        I1["Input: videot1.py"]:::input
        I2["Input: videot1_multigpu.py"]:::input
        I3["Input: videot1_woimg.py"]:::input
    end

    %% Video Generation Pipeline
    subgraph "Video Generation Pipeline"
        PM["Pipeline Module"]:::pipeline
        PM --> PF["Pyramid-Flow Engine"]:::engine
    end

    %% Pyramid-Flow Engine Subcomponents
    subgraph "Pyramid-Flow Engine"
        subgraph "Diffusion Schedulers"
            DDPM["Diffusion Scheduler [Cosine DDPM]"]:::engine
            FM["Diffusion Scheduler [Flow Matching]"]:::engine
        end
        PDiT["Pyramid DiT Module"]:::engine
        VAE["Video VAE Module"]:::engine
        %% Connect diffusion schedulers to Pyramid DiT and VAE for simplicity
        DDPM --> PDiT
        FM --> PDiT
        PDiT --> VAE
    end

    %% Test-Time Scaling (TTS) Module
    TTS["Test-Time Scaling (TTS)"]:::pipeline
    TTS_R["RandomLinearSearch"]:::pipeline
    TTS_T["TreeOfFramesSearch"]:::pipeline
    TTS --> TTS_R
    TTS --> TTS_T

    %% Final Output
    OUT["Final Video Output"]:::output

    %% Reward and Guidance Module
    REW["Reward Module (VisionReward)"]:::reward

    %% Trainer and Utility Modules
    TRN["Trainer Module"]:::trainer
    HPT["Hierarchical Prompts Utility"]:::trainer

    %% Scripts and Multi-GPU Support
    subgraph "Multi-GPU Support"
        MGPU["Multi-GPU Orchestrator"]:::script
        GPU0["GPU0: Reward Model"]:::gpu
        GPU1["GPU1: Pyramid-Flow"]:::gpu
        GPU2["GPU2: Image-CoT"]:::gpu
        GPU3["GPU3: LM"]:::gpu
        MGPU --> GPU0
        MGPU --> GPU1
        MGPU --> GPU2
        MGPU --> GPU3
    end

    %% Data Flow Connections
    I1 --- PM
    I2 --- PM
    I3 --- PM

    PF --> TTS
    TTS_R --> OUT
    TTS_T --> OUT

    PF --> REW
    REW -.-> PM

    %% Trainer and Utility branch (side process)
    PM --- TRN
    TRN --- HPT

    %% Click Events for Input/Interface Layer
    click I1 "https://github.com/liuff19/video-t1/blob/main/videot1.py"
    click I2 "https://github.com/liuff19/video-t1/blob/main/videot1_multigpu.py"
    click I3 "https://github.com/liuff19/video-t1/blob/main/videot1_woimg.py"

    %% Click Event for Video Generation Pipeline
    click PM "https://github.com/liuff19/video-t1/blob/main/pipeline/videot1_pipeline.py"

    %% Click Events for Pyramid-Flow Engine components
    click DDPM "https://github.com/liuff19/video-t1/blob/main/pyramid_flow/diffusion_schedulers/scheduling_cosine_ddpm.py"
    click FM "https://github.com/liuff19/video-t1/blob/main/pyramid_flow/diffusion_schedulers/scheduling_flow_matching.py"
    click PDiT "https://github.com/liuff19/video-t1/blob/main/pyramid_flow/pyramid_dit/pyramid_dit_for_video_gen_pipeline.py"
    click VAE "https://github.com/liuff19/video-t1/tree/main/pyramid_flow/video_vae/"

    %% Click Events for Reward Module
    click REW "https://github.com/liuff19/video-t1/blob/main/utils/visionreward.py"
    %% Additional resource in vqa folder
    click REW "https://github.com/liuff19/video-t1/blob/main/vqa/VisionReward_video_qa_select.txt"

    %% Click Events for Trainer/Utility Modules
    click TRN "https://github.com/liuff19/video-t1/tree/main/pyramid_flow/trainer_misc/"
    click HPT "https://github.com/liuff19/video-t1/blob/main/utils/hierarchical_prompts.py"

    %% Click Event for Multi-GPU Support
    click MGPU "https://github.com/liuff19/video-t1/blob/main/pyramid_flow/scripts/app_multigpu_engine.py"

    %% Styles
    classDef input fill:#add8e6,stroke:#000,stroke-width:1px;
    classDef pipeline fill:#98fb98,stroke:#000,stroke-width:1px;
    classDef engine fill:#87cefa,stroke:#000,stroke-width:1px;
    classDef reward fill:#ffb6c1,stroke:#000,stroke-width:1px;
    classDef trainer fill:#90ee90,stroke:#000,stroke-width:1px;
    classDef script fill:#ffa500,stroke:#000,stroke-width:1px;
    classDef gpu fill:#dda0dd,stroke:#000,stroke-width:1px;
    classDef output fill:#ffffe0,stroke:#000,stroke-width:1px;

    %% Connect main sections
    I1 & I2 & I3 --> PM
    PF --> TTS
    TTS_R & TTS_T --> OUT
    PF --> REW
    REW -.-> PM
    PM --- TRN
    TRN --- HPT
    MGPU --> GPU0
    MGPU --> GPU1
    MGPU --> GPU2
    MGPU --> GPU3

    %% Connect Pipeline to Multi-GPU Support (informative association)
    PM --- MGPU
```
