How liuff19/video-t1 works
Main Components
• Input/Interface Layer: This is where the user provides a text prompt. In the code, this is most clearly shown in files like videot1.py, videot1_multigpu.py, and videot1_woimg.py.
• Video Generation Pipeline:
• Pipeline Module: The pipeline folder (e.g., pipeline/videot1_pipeline.py) sets up the end-to-end video generation process.
• Pyramid-Flow Engine: The pyramid_flow directory is the core of the video generation engine. It contains several sub-modules:
▪ Diffusion Schedulers: Files in pyramid_flow/diffusion_schedulers govern various denoising strategies (e.g., scheduling_cosine_ddpm.py, scheduling_flow_matching.py).
▪ Pyramid DiT: The pyramid_flow/pyramid_dit module implements Transformer-like models (e.g., flux_modules and mmdit_modules) for embedding, normalization, and the pyramid design.
▪ Video VAE: Located in pyramid_flow/video_vae, these modules implement VAE and discriminator networks which focus on the video generation’s temporal dynamics.
• Reward and Guidance Module: A VisionReward model is used (see references in the README and vqa folder for some related data) to provide feedback. This is coupled with the optional Image-CoT generation (image branch) and hierarchical prompts (supported by additional scripts and checkpoints that need to be loaded).
• Trainer/Utility Modules: The pyramid_flow/trainer_misc contains supportive functionalities for training and model communication (e.g., fsdp_trainer.py, vae_ddp_trainer.py). Also, the utils folder holds helper functions like hierarchical_prompts.
• Scripts and Multi-GPU Support: The pyramid_flow/scripts contains scripts (such as app_multigpu_engine.py) and shell scripts for training and inference workflows; videot1_multigpu.py enables the loading of different models on distinct GPUs.
• External Dependencies and Checkpoints: The README instructs downloading external checkpoints for Pyramid-Flow, VisionReward, Image-CoT, and optionally hierarchical prompt models (DeepSeek-R1-Distill-Llama-8B).
• In the Video Generation Pipeline, represent sub-components:
• Data Preprocessing (if present, implied by initial prompt handling).
• Pyramid-Flow Engine – include separate blocks for:
– Diffusion Schedulers (illustrate the two types: cosine DDPM and flow matching).
– Pyramid DiT blocks split into Flux (for one branch of processing) and MMDiT modules (for an alternative branch or modification).
– Video VAE module for the temporal synthesis part.
• Route the outputs of the Pyramid-Flow Engine into the “Test-Time Scaling” module which refines the result by employing TTS strategies (as described: random linear search and Tree of Frames search with three stages).
Architectural Patterns and Key Details
• MVC-like separation: Input → Processing (Pipeline with distinct modules) → Evaluation (Reward feedback) → Inference Output.
• Modular design: Each sub-module (diffusion schedulers, pyramid_dit, video_vae, etc.) encapsulates a specific stage in the generation process.
• Underline the importance of test-time iterative strategies: The TTS approach uses a combination of linear sampling and hierarchical tree strategies for evaluating video generation quality.