How lihaoyun6/comfyui-seedvr2_videoupscaler works
1. What this repository is
This is a ComfyUI custom node package for SeedVR2, with a second execution path for standalone CLI inference. Both modes ultimately run the same internal upscale pipeline; the difference is only the frontend and how inputs/outputs are wired.
2. Main entry points
The two user-facing entry points are src/interfaces/comfyui_node.py for ComfyUI integration and inference_cli.py for command-line batch processing. The README also ships example ComfyUI workflows for image and video upscaling, which show the intended graph-level integration.
3. Core orchestration layer
src/core/generation.py, src/core/infer.py, and src/core/model_manager.py are the main control plane. They handle model selection, lazy loading/unloading, inference execution, and output assembly. This is the layer that ties together preprocessing, model execution, memory policy, and final export.
4. Input flow and preprocessing
The data path begins with video or image frames from either ComfyUI or the CLI. Frame handling and spatial normalization live under src/data/image/transforms, including resize and crop utilities. These transforms prepare frames to the model’s expected dimensions and aspect-ratio rules before inference.
5. Model families
The repository contains three major model subsystems: SeedVR2 DiT models in src/models/dit, a newer variant in src/models/dit_v2, and the video VAE stack in src/models/video_vae_v3. The DiT modules implement the diffusion transformer backbone; the VAE modules handle video latent encoding/decoding; and the two DiT trees suggest an evolution path or compatibility split between model generations.
6. Diffusion runtime
Shared diffusion logic lives in src/common/diffusion, including samplers, schedules, and timestep utilities. This layer is model-agnostic and provides the numerical machinery used during generation, while src/common/diffusion/config.py and related types keep the runtime configuration separated from model definitions.
7. Memory and performance boundary
src/optimization is a dedicated subsystem for low-VRAM execution and performance tuning. It includes BlockSwap support, memory management, GGUF dequantization/ops, compatibility helpers, and performance utilities. This is the main boundary where the repo trades speed for memory savings, especially important because the README emphasizes high VRAM usage and optional CPU offload.
8. Distributed and multi-GPU support
src/common/distributed contains the multi-device execution helpers used by standalone mode. The CLI supports a comma-separated CUDA device list, so the pipeline can shard work across GPUs. This is a separate execution path from ComfyUI’s typical single-process node flow.
9. Model acquisition and registry
src/utils/downloads.py and src/utils/model_registry.py support automatic model discovery and fetching from external model sources, primarily Hugging Face as described in the README. The repo also includes local config files for 3B and 7B variants, plus embedded prompt/negative embeddings (pos_emb.pt, neg_emb.pt), so model selection is driven by local registry/config rather than hardcoded paths.
10. Common infrastructure
The src/common package holds reusable plumbing: config parsing, caching, logging, seeding, decorators, partitioning, and half-precision fixes. This is the shared foundation used by both the ComfyUI node and the CLI to keep behavior consistent across execution modes.
11. ComfyUI workflow integration
The example JSON workflows show the repo is meant to be embedded in a node graph with optional BlockSwap configuration. The output boundary is therefore node-to-node rather than service-to-service: frames enter the SeedVR2 node, are processed internally, and return as upscaled video frames or image sequences for downstream ComfyUI nodes.
12. Output path
The pipeline can emit either video or PNG sequences, as shown by the standalone CLI options. The final stage is responsible for reassembling processed frames into the requested format and writing to the chosen output path, making export format a configurable terminal boundary rather than part of the model core.
13. Operational characteristics
The architecture is deliberately GPU-centric and memory constrained. The README’s warnings, batch-size guidance, and BlockSwap options reflect the main runtime reality: inference is dominated by VRAM management, not just compute. That makes memory policy an architectural concern, not just an optimization detail.
14. Repository shape for diagramming
If you draw this architecture, the cleanest split is: user entry points on the left, orchestration in the middle, shared runtime and model families below, and optimization/distributed support as side-car subsystems. External dependencies to show are ComfyUI, CUDA/PyTorch, local model files, and Hugging Face for automatic model download.
Main entry points: generation.py Generation · infer.py Infer · model_manager.py Model manager · comfyui_node.py ComfyUI node
Components 19
Entry points
User-facing ways to run the pipeline
- ComfyUI node · ComfyUI custom node entry point
src/interfaces/comfyui_node.py - CLI runner · Standalone batch inference entry point
inference_cli.py
Orchestration
Control plane that prepares inputs, runs inference, and assembles outputs
- Generation · Top-level pipeline orchestration and output assembly
src/core/generation.py - Infer · Inference execution and runtime coordination
src/core/infer.py - Model manager · Model selection, lazy loading, and unloading
src/core/model_manager.py - Image transforms · Resize and crop utilities for frame normalization
src/data/image/transforms
Runtime & models
Shared diffusion runtime and model families
- Diffusion config · Shared diffusion runtime configuration and types
src/common/diffusion/config.py - Diffusion utils · Samplers, schedules, timestep utilities, and shared diffusion helpers
src/common/diffusion - DiT · SeedVR2 diffusion transformer backbone
src/models/dit - DiT v2 · Updated diffusion transformer variant
src/models/dit_v2 - Video VAE · Video latent encoder/decoder stack
src/models/video_vae_v3
Support systems
Memory, distribution, registry, and example integration assets
- BlockSwap · Low-VRAM execution and module swapping
src/optimization/blockswap.py - Memory manager · VRAM management and allocation helpers
src/optimization/memory_manager.py - Distributed · Multi-device execution helpers for standalone mode
src/common/distributed - Model registry · Model discovery and download coordination
src/utils/model_registry.py - Downloads · External model fetching helpers
src/utils/downloads.py - 3B config · Configuration for the smaller model variant
configs_3b/main.yaml - 7B config · Configuration for the larger model variant
configs_7b/main.yaml - Example workflows · ComfyUI example graphs for image and video upscaling
example_workflows
Connections 20
- ComfyUI node → Generation: invokes · ComfyUI node delegates execution into the shared pipeline
- CLI runner → Generation: invokes · CLI uses the same orchestration path as the node frontend
- Generation → Image transforms: preprocess · Frames are normalized before model execution
- Generation → Model manager: selects model · Orchestration asks for the appropriate model variant
- Generation → Infer: runs · Generation coordinates the actual inference pass
- Generation → BlockSwap: applies memory policy · Low-VRAM behavior is wired into the pipeline
- Generation → Distributed: shards · Standalone mode can spread work across GPUs
- Infer → Diffusion utils: uses · Inference depends on shared samplers, schedules, and timestep logic
- Infer → Diffusion config: reads config · Runtime parameters are separated from model definitions
- Infer → DiT: executes · One supported backbone family
- Infer → DiT v2: executes · Alternative backbone family
- Infer → Video VAE: encodes/decodes · Video latent codec participates in the pipeline
- Model manager → Model registry: resolves · Model lookup is driven by registry metadata
- Model manager → Downloads: fetches · Missing models can be acquired automatically
- Model manager → 3B config: loads · One model family configuration
- Model manager → 7B config: loads · Another model family configuration
- Model registry → Downloads: requests · Registry-backed acquisition can trigger downloads
- Example workflows → ComfyUI node: documents · Example workflows show how the custom node is embedded
- BlockSwap → Memory manager: coordinates · Block swapping relies on shared memory controls
- Distributed → Infer: supports · Multi-GPU helpers extend the standalone inference path