How picsart-ai-research/streamingt2v works
StreamingSVD turns input images into long, temporally consistent videos through autoregressive image-to-video generation, followed by video enhancement (including optional randomized blending) and frame interpolation. The inference entry point prepares the input and components; the StreamingSVD trainer owns generation, while enhancement and interpolation are distinct downstream capabilities. The graph also shows the model-loading and diffusion-model layers. Some implementation excerpts are partial, so uncertain runtime links are marked dashed with no evidence path.
Main entry points: inference_i2v.py Inference pipeline · streaming_svd.py StreamingSVD trainer · i2v_enhance_interface.py Enhancement interface · unet_3d_blocks.py 3D UNet blocks
Components 20
Inference
- Inference pipeline
code/inference_i2v.py - Image dataset
code/dataloader/single_image_dataset.py - Video data module
code/dataloader/video_data_module.py
Video generation
- Module loader
code/modules/loader/module_loader.py - StreamingSVD trainer
code/diffusion_trainer/streaming_svd.py
Model internals
- Video diffusion model
code/models/diffusion/video_model.py - Diffusion UNet
code/models/svd/sgm/modules/diffusionmodules/openaimodel.py - Video attention
code/models/svd/sgm/modules/video_attention.py - Attention layers
code/models/svd/sgm/modules/attention.py - Video autoencoder
code/models/svd/sgm/modules/autoencoding/temporal_ae.py
Enhancement
- Enhancement interface
code/i2v_enhance/i2v_enhance_interface.py - I2VGen-XL pipeline
code/i2v_enhance/pipeline_i2vgen_xl.py - Enhancement UNet
code/i2v_enhance/unet_i2vgen_xl.py - 3D UNet blocks
code/i2v_enhance/unet_3d_blocks.py - Spatial transformer
code/i2v_enhance/transformer_2d.py - Temporal transformer
code/i2v_enhance/transformer_temporal.py - Frame interpolation
code/i2v_enhance/thirdparty/VFI/Trainer.py
Output
- Result processing
code/utils/result_processor.py
Other components
- Image input
- Generated video
Connections 23
- Image input → Inference pipeline: provides image
README.md - Inference pipeline → Image dataset: creates dataset
code/inference_i2v.py - Inference pipeline → Video data module: configures input
code/inference_i2v.py - Video data module → Image dataset: uses factory
code/dataloader/video_data_module.py - Inference pipeline → StreamingSVD trainer: runs prediction
code/inference_i2v.py - StreamingSVD trainer → Module loader: uses loader
code/diffusion_trainer/streaming_svd.py - StreamingSVD trainer → Video diffusion model: runs diffusion
- Video diffusion model → Diffusion UNet: uses backbone
code/models/diffusion/video_model.py - Diffusion UNet → Video attention: uses attention
code/models/svd/sgm/modules/diffusionmodules/openaimodel.py - Video attention → Attention layers: uses layers
code/models/svd/sgm/modules/video_attention.py - StreamingSVD trainer → Video autoencoder: decodes frames
code/diffusion_trainer/streaming_svd.py - Inference pipeline → Enhancement interface: initializes enhancement
code/inference_i2v.py - Enhancement interface → I2VGen-XL pipeline: initializes pipeline
code/i2v_enhance/i2v_enhance_interface.py - I2VGen-XL pipeline → Enhancement UNet: runs enhancement
code/i2v_enhance/pipeline_i2vgen_xl.py - Enhancement UNet → 3D UNet blocks: uses blocks
code/i2v_enhance/unet_i2vgen_xl.py - 3D UNet blocks → Spatial transformer: uses transformer
code/i2v_enhance/unet_3d_blocks.py - 3D UNet blocks → Temporal transformer: uses temporal layers
code/i2v_enhance/unet_3d_blocks.py - Spatial transformer → Temporal transformer: uses temporal layers
code/i2v_enhance/transformer_temporal.py - Inference pipeline → Frame interpolation: initializes interpolation
code/inference_i2v.py - StreamingSVD trainer → Enhancement interface: enhances generated video
README.md - Enhancement interface → Frame interpolation: interpolates frames
code/i2v_enhance/i2v_enhance_interface.py - Inference pipeline → Result processing: processes results
- Result processing → Generated video: saves video