# yiboli-4110/environment-controllable-video-generation architecture

> Architecture diagram of the GitHub repository yiboli-4110/environment-controllable-video-generation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yiboli-4110/environment-controllable-video-generation
- Repository: https://github.com/yiboli-4110/environment-controllable-video-generation

## Overview

DiffSynth is a PyTorch toolkit for generative image and video workflows; the README foregrounds environment-controllable video generation and synthetic training data. The principal runtime path is a user-supplied prompt and/or video/control inputs, model loading, pipeline inference, then video output. The tree also exposes training, model-family, control, and postprocessing subsystems. Source excerpts are partial, so links among unsampled pipeline internals are omitted rather than inferred.

Main entry points: [`diffsynth/pipelines/pipeline_runner.py`](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/pipelines/pipeline_runner.py) (Video runner), [`diffsynth/pipelines`](https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/pipelines) (Generation pipelines), [`diffsynth/models`](https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/models) (Model families), [`diffsynth/models/model_manager.py`](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/models/model_manager.py) (Model manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Inputs and prompts"]
  node_prompters["Prompt handling"]
  node_data["Image/video data"]
end

subgraph group_runtime["Generation runtime"]
  node_runner["Video runner<br/>[pipeline_runner.py]"]
  node_pipelines["Generation pipelines"]
  node_base["Pipeline runtime<br/>[__init__.py]"]
  node_schedulers["Denoising schedulers"]
  node_parallel["Context parallelism"]
  node_vram["VRAM management<br/>[layers.py]"]
end

subgraph group_models["Models and controls"]
  node_manager["Model manager<br/>[model_manager.py]"]
  node_downloader["Model downloader<br/>[downloader.py]"]
  node_model_families["Model families"]
  node_controlnets["ControlNet units<br/>[controlnet_unit.py]"]
  node_camera["Camera control"]
  node_motion["Motion control"]
end

subgraph group_training["Training and processing"]
  node_processors["Video processors"]
  node_trainers["Training workflows"]
  node_dataset["Dataset operators<br/>[unified_dataset.py]"]
  node_lora["LoRA support"]
end

node_user(("User"))
node_output["Generated video"]

node_user -->|"provides inputs"| node_runner
node_runner -->|"loads video"| node_data
node_runner -->|"loads models"| node_manager
node_runner -->|"constructs and runs"| node_pipelines
node_runner -->|"loads smoother"| node_processors
node_runner -->|"saves frames/video"| node_data
node_runner -->|"writes result"| node_output
node_manager -->|"downloads models"| node_downloader
node_manager -->|"loads model classes"| node_model_families
node_pipelines -->|"generates"| node_output
node_prompters -->|"supplies prompt features"| node_pipelines
node_data -->|"supplies media"| node_pipelines
node_controlnets -->|"conditions generation"| node_pipelines
node_camera -->|"conditions camera"| node_pipelines
node_motion -->|"conditions motion"| node_pipelines
node_schedulers -->|"steps denoising"| node_pipelines
node_parallel -->|"parallelizes inference"| node_model_families
node_vram -->|"manages memory"| node_model_families
node_dataset -->|"prepares samples"| node_trainers
node_trainers -->|"trains models"| node_model_families
node_lora -->|"adapts weights"| node_model_families

click node_prompters "https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/prompters"
click node_data "https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/data"
click node_runner "https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/pipelines/pipeline_runner.py"
click node_pipelines "https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/pipelines"
click node_base "https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/utils/__init__.py"
click node_manager "https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/models/model_manager.py"
click node_downloader "https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/models/downloader.py"
click node_model_families "https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/models"
click node_controlnets "https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/controlnets/controlnet_unit.py"
click node_camera "https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/models/wan_video_camera_controller.py"
click node_motion "https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/models/wan_video_motion_controller.py"
click node_schedulers "https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/schedulers"
click node_parallel "https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/distributed/xdit_context_parallel.py"
click node_vram "https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/vram_management/layers.py"
click node_processors "https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/processors"
click node_trainers "https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/trainers"
click node_dataset "https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/trainers/unified_dataset.py"
click node_lora "https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/lora"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_prompters,node_data,node_user toneBlue
class node_runner,node_pipelines,node_base,node_schedulers,node_parallel,node_vram toneAmber
class node_manager,node_downloader,node_model_families,node_controlnets,node_camera,node_motion toneMint
class node_processors,node_trainers,node_dataset,node_lora toneRose
class node_output toneIndigo
```

## Components

### Inputs and prompts

- [Prompt handling](https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/prompters): `diffsynth/prompters`
- [Image/video data](https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/data): `diffsynth/data`

### Generation runtime

- [Video runner](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/pipelines/pipeline_runner.py): `diffsynth/pipelines/pipeline_runner.py`
- [Generation pipelines](https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/pipelines): `diffsynth/pipelines`
- [Pipeline runtime](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/utils/__init__.py): `diffsynth/utils/__init__.py`
- [Denoising schedulers](https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/schedulers): `diffsynth/schedulers`
- [Context parallelism](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/distributed/xdit_context_parallel.py): `diffsynth/distributed/xdit_context_parallel.py`
- [VRAM management](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/vram_management/layers.py): `diffsynth/vram_management/layers.py`

### Models and controls

- [Model manager](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/models/model_manager.py): `diffsynth/models/model_manager.py`
- [Model downloader](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/models/downloader.py): `diffsynth/models/downloader.py`
- [Model families](https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/models): `diffsynth/models`
- [ControlNet units](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/controlnets/controlnet_unit.py): `diffsynth/controlnets/controlnet_unit.py`
- [Camera control](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/models/wan_video_camera_controller.py): `diffsynth/models/wan_video_camera_controller.py`
- [Motion control](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/models/wan_video_motion_controller.py): `diffsynth/models/wan_video_motion_controller.py`

### Training and processing

- [Video processors](https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/processors): `diffsynth/processors`
- [Training workflows](https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/trainers): `diffsynth/trainers`
- [Dataset operators](https://github.com/yiboli-4110/environment-controllable-video-generation/blob/main/diffsynth/trainers/unified_dataset.py): `diffsynth/trainers/unified_dataset.py`
- [LoRA support](https://github.com/yiboli-4110/environment-controllable-video-generation/tree/main/diffsynth/lora): `diffsynth/lora`

### Other components

- **User**
- **Generated video**

## Connections

- User → Video runner: provides inputs
- Video runner → Image/video data: loads video
- Video runner → Model manager: loads models
- Video runner → Generation pipelines: constructs and runs
- Video runner → Video processors: loads smoother
- Video runner → Image/video data: saves frames/video
- Video runner → Generated video: writes result
- Model manager → Model downloader: downloads models
- Model manager → Model families: loads model classes
- Generation pipelines → Generated video: generates
- Prompt handling → Generation pipelines: supplies prompt features
- Image/video data → Generation pipelines: supplies media
- ControlNet units → Generation pipelines: conditions generation
- Camera control → Generation pipelines: conditions camera
- Motion control → Generation pipelines: conditions motion
- Denoising schedulers → Generation pipelines: steps denoising
- Context parallelism → Model families: parallelizes inference
- VRAM management → Model families: manages memory
- Dataset operators → Training workflows: prepares samples
- Training workflows → Model families: trains models
- LoRA support → Model families: adapts weights