How kijai/comfyui-dsine works
1. Repository purpose
- This is a ComfyUI integration for DSINE, a surface-normal estimation model.
- The repo combines two layers: the core DSINE inference/model code and a ComfyUI node wrapper that exposes it inside a workflow UI.
2. Top-level architecture
- The most important boundary is between:
- UI/runtime integration: nodes.py and examples/DSINE_example_workflow_01.json
- Model/inference implementation: models/, utils/, and test.py
- Documentation and visual assets live separately under docs/, and are not part of the runtime path.
3. ComfyUI entry point
- nodes.py is the integration surface for ComfyUI.
- This file is where the model is likely registered as a node, inputs/outputs are defined, and inference is triggered from the graph.
- In an architecture diagram, treat this as the adapter layer between ComfyUI and the DSINE model code.
4. Model subsystem
- models/dsine.py contains the main DSINE network definition and forward logic.
- models/submodules.py holds reusable building blocks used by the model, such as intermediate layers, encoders/decoders, or helper modules.
- This subsystem is the core compute layer: it takes an image plus camera geometry and produces surface normals.
5. Geometry and rotation utilities
- utils/rotation.py likely contains the math for converting between rotations, vectors, and normal representations.
- This is important because DSINE’s method explicitly models relative rotation between neighboring surface normals.
- In the architecture, this sits between the neural network outputs and the geometric interpretation of normals.
6. General utility layer
- utils/utils.py appears to house shared helpers such as image preprocessing, file/path handling, intrinsics loading, checkpoint loading, or tensor conversion utilities.
- The README indicates the model can use either provided camera intrinsics or approximate intrinsics from a default field of view, so this utility layer likely handles that camera-parameter path.
7. Standalone inference path
- test.py is the repo’s command-line evaluation/inference entry point.
- It reads sample images from samples/img/, loads any matching .txt intrinsics files, runs the DSINE model, and writes results to samples/output/.
- This script is the clearest end-to-end dataflow example outside ComfyUI.
8. Input/output data flow
- The expected inference flow is:
- image file in samples/img/
- optional camera intrinsics from sibling .txt file
- preprocessing and tensor conversion
- DSINE model inference
- predicted surface-normal output saved to an output directory
- The architecture is therefore image + camera geometry in, normal map out.
9. Camera model dependency
- Camera intrinsics are a first-class part of the pipeline, not optional metadata.
- The README says performance improves with known intrinsics, but the code can fall back to approximate values using a 60° FOV assumption.
- That makes camera geometry a distinct subsystem in the architecture, adjacent to preprocessing and model execution.
10. Examples and workflow packaging
- examples/DSINE_example_workflow_01.json is a ComfyUI workflow example showing how the node is wired in practice.
- This file is not core logic, but it is useful for understanding the node’s expected inputs/outputs and how it fits into a broader graph.
11. Documentation and presentation assets
- docs/ contains the original project’s static documentation site assets: HTML, CSS, JS, images, and comparison visuals.
- paper.pdf is included for reference, but it is not part of the runtime system.
- These assets support understanding and validation, but do not participate in inference.
12. External dependencies and runtime boundaries
- Runtime depends on PyTorch and likely additional model-specific packages listed in requirements.txt.
- The README also mentions downloading pretrained weights into ./checkpoints/, which is an external artifact dependency separate from source control.
- So the deployable system is split across source code + local checkpoint files + input images/intrinsics.
13. What to draw in an architecture diagram
- A good diagram would show:
- ComfyUI graph / node runtime
- nodes.py as the adapter
- preprocessing + intrinsics handling
- DSINE model core in models/
- geometry helpers in utils/rotation.py
- output surface-normal maps
- optional standalone test.py inference path
- external checkpoint weights as a separate dependency
Main entry points: utils.py Shared utils · nodes.py ComfyUI node · dsine.py DSINE model · test.py Inference script
Components 14
Runtime integration
ComfyUI adapter and standalone inference entrypoints.
- ComfyUI node · Registers DSINE inside ComfyUI and bridges workflow inputs to model inference.
nodes.py - Inference script · Standalone image evaluation path that loads inputs, runs DSINE, and writes outputs.
test.py - Package init · Repository package entry point for Python importability.
__init__.py
DSINE core
Neural network and reusable building blocks for surface-normal estimation.
- DSINE model · Main surface-normal estimation network and forward pass logic.
models/dsine.py - Model blocks · Reusable internal layers and components used by the DSINE network.
models/submodules.py
Geometry utils
Rotation and normal-representation helpers.
- Rotation math · Conversions between rotations, vectors, and normal representations.
utils/rotation.py - Shared utils · Common helpers for preprocessing, camera parameters, checkpoint loading, and tensor/file handling.
utils/utils.py
Inputs and outputs
Sample images, intrinsics sidecars, and generated results.
- Sample images · Example input images used by the standalone inference path.
samples/img - Intrinsics files · Optional per-image .txt camera intrinsics accompanying sample images.
samples/img/office_01.txt - Output dir · Destination for predicted surface-normal results from the CLI path.
samples/output - Checkpoints · Downloaded pretrained DSINE weights used at runtime but stored outside source control.
Docs and examples
Workflow example and static documentation assets.
- Workflow example · Example ComfyUI graph showing how the DSINE node is wired.
examples/DSINE_example_workflow_01.json - Docs site · Static project documentation, visuals, and comparison assets.
docs/index.html - Project README · Primary description of setup, usage, inputs, and model behavior.
README.md
Connections 14
- ComfyUI node → DSINE model: runs inference · ComfyUI node delegates prediction to the DSINE network.
- ComfyUI node → Shared utils: preprocesses · Adapter uses shared helpers for input conversion and runtime setup.
- ComfyUI node → Rotation math: interprets normals · Geometry helpers support normal/rotation handling at the adapter boundary.
- Inference script → Shared utils: loads inputs · CLI script uses shared helpers for image, intrinsics, and checkpoint handling.
- Inference script → DSINE model: executes · Standalone inference script instantiates and runs the DSINE model.
- DSINE model → Model blocks: uses · Core network composes reusable model submodules.
- DSINE model → Rotation math: outputs normals · Model predictions are converted through rotation/normal math.
- Shared utils → Checkpoints: loads weights · Utility layer resolves pretrained model artifacts from checkpoint storage.
- Shared utils → Intrinsics files: reads camera data · Utility layer can parse per-image intrinsics sidecars.
- Shared utils → Sample images: ingests · Preprocessing starts from sample or user-provided input images.
- Inference script → Output dir: writes results · Predicted normal maps are saved to the output directory.
- Workflow example → ComfyUI node: demonstrates · Example workflow shows the ComfyUI node in context.
- Docs site → Project README: documents · Static docs and README together explain usage and results.
- Package init → ComfyUI node: exposes · Package entry point makes the ComfyUI integration importable.