How kijai/comfyui-wananimatepreprocess works
1. What this repo is
- This is a ComfyUI custom nodes package for the preprocessing stage of Wan Video 2.2 Animate.
- Its job is not to generate video itself, but to prepare pose- and face-related conditioning data: whole-body keypoints, face crops, and keypoint lists suitable for downstream SAM2 segmentation or related animation pipelines.
2. Top-level architecture
- The code is organized around a node layer that ComfyUI loads, plus several utility/model wrapper layers underneath.
- The main visible boundaries are:
- ComfyUI node registration and orchestration
- ONNX model loading/inference
- Pose/keypoint conversion and visualization
- Retargeting / preprocessing helpers
- Example workflows and packaging
3. ComfyUI integration boundary
- __init__.py and nodes.py are the entry points for ComfyUI.
- nodes.py is likely where the custom node classes are defined and exposed to the UI.
- This is the boundary where ComfyUI inputs/outputs are translated into the repo’s internal pose-processing functions.
4. Inference subsystem
- models/onnx_models.py and onetoall/infer_function.py form the model execution layer.
- The README makes it clear the repo uses ONNX models, including:
- YOLOv10m ONNX for detection
- ViTPose ONNX for whole-body pose estimation
- This layer is responsible for loading model artifacts from disk and running inference on images/frames.
5. Model artifact boundary
- The repository does not vendor the model weights; they are external artifacts placed under ComfyUI/models/detection.
- This matters architecturally because the code expects a local model cache / filesystem boundary rather than an API service.
- The huge ViTPose model is split into two files (.onnx plus .bin) to work around ONNX size limits, so the model loader likely needs to handle that split format.
6. Pose processing subsystem
- pose_utils/pose2d_utils.py and pose_utils/human_visualization.py are the pose-specific helpers.
- Based on their names and the README, these utilities likely:
- normalize or transform 2D keypoints
- map model outputs into a consistent human-pose schema
- render skeletons / overlays for debugging or preview
- This is the layer that turns raw detector output into something usable by downstream nodes.
7. Retargeting / preprocessing logic
- retarget_pose.py and the top-level utils.py suggest higher-level preprocessing steps built on top of raw keypoints.
- In this repo’s context, “retargeting” likely means transforming pose data from detected people into a form suitable for animation transfer or conditioning.
- This sits between inference and the final exported conditioning data.
8. One-to-all support code
- The onetoall/ package (infer_function.py, utils.py) appears to be a reusable helper module for inference-time operations.
- Architecturally, it likely contains glue code for:
- batching or dispatching model calls
- image/array preprocessing
- postprocessing model outputs into standardized formats
- It acts as a shared service layer under the nodes, rather than as a user-facing component.
9. Example workflow boundary
- example_workflows/WanAnimate_native_example_01.json shows how the nodes are expected to be wired in ComfyUI.
- This is important for diagramming because it reveals the data flow expected by the package: input media → detection/pose extraction → crops/keypoints → downstream animation pipeline.
- The example workflow is effectively a reference integration, not core runtime logic.
10. External dependencies and runtime
- The repo is Python-based and packaged with pyproject.toml and requirements.txt.
- Its runtime dependencies are driven by:
- ComfyUI as the host environment
- ONNX / ONNX Runtime-style model execution
- likely image processing and numerical libraries used for pose decoding and visualization
- The architecture assumes local execution with GPU/accelerated inference where available, but the repo itself is not a service backend.
11. Operational output of the pipeline
- The practical outputs of the repo are:
- whole-body pose keypoints
- face crops
- visualized pose overlays
- keypoint lists that can feed segmentation or animation modules
- These outputs are passed onward to other ComfyUI nodes or to downstream Wan Animate preprocessing steps.
12. Packaging and release flow
- .github/workflows/publish.yml indicates the repo is packaged and published through GitHub Actions.
- That means the architectural deployment boundary is a ComfyUI extension install/update workflow, not a standalone application deployment.
13. How to draw the diagram
- Draw the center as a ComfyUI custom node extension.
- Put external model files on disk as a separate dependency box.
- Show the main flow as:
- input image/frame from ComfyUI
- detection and ViTPose ONNX inference
- pose postprocessing / visualization / retargeting
- output keypoints and crops to downstream nodes
- Keep pose_utils, onetoall, and models as internal support subsystems behind the node layer.
Main entry points: nodes.py Nodes · infer_function.py Infer Func · pose2d_utils.py Pose2D · retarget_pose.py Retarget
Components 12
ComfyUI Extension
Custom node entrypoints and orchestration layer.
Inference Stack
ONNX model loading and runtime execution.
- ONNX Loader · Loads detector and pose ONNX artifacts from disk.
models/onnx_models.py - Infer Func · Runs image preprocessing, model dispatch, and postprocessing.
onetoall/infer_function.py - O2A Utils · Shared utility layer for inference-time array and image handling.
onetoall/utils.py
Pose Processing
Keypoint normalization, visualization, and retargeting.
- Pose2D · Transforms and standardizes 2D pose keypoints.
pose_utils/pose2d_utils.py - Visualize · Draws skeletons and pose previews for debugging.
pose_utils/human_visualization.py - Retarget · Converts detected pose data into animation-ready conditioning.
retarget_pose.py - Utils · General-purpose preprocessing helpers used by pose flow.
utils.py
Packaging & Examples
Distribution metadata and reference workflow.
- Example Flow · Reference graph showing the intended node wiring.
example_workflows/WanAnimate_native_example_01.json - Package Meta · Packaging and dependency metadata for the extension.
pyproject.toml
Model Cache
External model artifacts expected on local disk.
- Model Files · External YOLOv10m and ViTPose ONNX assets placed under the ComfyUI model cache.
Connections 14
- Init → Nodes: loads · ComfyUI imports the node definitions through the package entrypoint.
- Nodes → Infer Func: calls · Node execution delegates preprocessing and inference work.
- Nodes → Pose2D: uses · Nodes convert raw results into the package pose schema.
- Nodes → Retarget: uses · Nodes expose retargeting/preprocessing operations.
- Infer Func → ONNX Loader: executes · Inference function loads and runs the ONNX models.
- Infer Func → O2A Utils: uses · Shared runtime helpers support preprocessing and batching.
- ONNX Loader → Model Files: reads · Model loader resolves artifacts from the local ComfyUI model directory.
- Pose2D → Visualize: renders · Keypoints are optionally visualized as overlays.
- Pose2D → Retarget: feeds · Normalized pose data drives retargeting transforms.
- Utils → Pose2D: supports · General helpers assist pose conversion and cleanup.
- Retarget → Utils: depends · Retargeting reuses shared preprocessing helpers.
- Example Flow → Nodes: demonstrates · Example workflow shows how the nodes are composed in ComfyUI.
- Package Meta → Nodes: packages · Build metadata defines the extension as a distributable package.
- Package Meta → Infer Func: depends on · Packaging captures runtime dependencies for inference.