How yiboli-4110/environment-controllable-video-generation works
DiffSynth is a PyTorch toolkit for generative image and video workflows; the README foregrounds environment-controllable video generation and synthetic training data. The principal runtime path is a user-supplied prompt and/or video/control inputs, model loading, pipeline inference, then video output. The tree also exposes training, model-family, control, and postprocessing subsystems. Source excerpts are partial, so links among unsampled pipeline internals are omitted rather than inferred.
Main entry points: pipeline_runner.py Video runner · pipelines Generation pipelines · models Model families · model_manager.py Model manager
Components 20
Inputs and prompts
- Prompt handling
diffsynth/prompters - Image/video data
diffsynth/data
Generation runtime
- Video runner
diffsynth/pipelines/pipeline_runner.py - Generation pipelines
diffsynth/pipelines - Pipeline runtime
diffsynth/utils/__init__.py - Denoising schedulers
diffsynth/schedulers - Context parallelism
diffsynth/distributed/xdit_context_parallel.py - VRAM management
diffsynth/vram_management/layers.py
Models and controls
- Model manager
diffsynth/models/model_manager.py - Model downloader
diffsynth/models/downloader.py - Model families
diffsynth/models - ControlNet units
diffsynth/controlnets/controlnet_unit.py - Camera control
diffsynth/models/wan_video_camera_controller.py - Motion control
diffsynth/models/wan_video_motion_controller.py
Training and processing
- Video processors
diffsynth/processors - Training workflows
diffsynth/trainers - Dataset operators
diffsynth/trainers/unified_dataset.py - LoRA support
diffsynth/lora
Other components
- User
- Generated video
Connections 21
- User → Video runner: provides inputs
- Video runner → Image/video data: loads video
- Video runner → Model manager: loads models
- Video runner → Generation pipelines: constructs and runs
- Video runner → Video processors: loads smoother
- Video runner → Image/video data: saves frames/video
- Video runner → Generated video: writes result
- Model manager → Model downloader: downloads models
- Model manager → Model families: loads model classes
- Generation pipelines → Generated video: generates
- Prompt handling → Generation pipelines: supplies prompt features
- Image/video data → Generation pipelines: supplies media
- ControlNet units → Generation pipelines: conditions generation
- Camera control → Generation pipelines: conditions camera
- Motion control → Generation pipelines: conditions motion
- Denoising schedulers → Generation pipelines: steps denoising
- Context parallelism → Model families: parallelizes inference
- VRAM management → Model families: manages memory
- Dataset operators → Training workflows: prepares samples
- Training workflows → Model families: trains models
- LoRA support → Model families: adapts weights