How yedp123/comfyui-yedp-action-director works
1. What this repo is
This is a ComfyUI custom node that embeds a full client-side 3D director/viewport into the workflow. The architecture is split between a small Python node package and a much larger browser-based scene editor/rendering engine.
2. Python/ComfyUI boundary
__init__.py and nodes.py are the ComfyUI integration layer. Their job is to register the node, accept/emit workflow parameters, and carry serialized scene state inside standard ComfyUI JSON workflows. The Python side appears lightweight; the heavy 3D work is not done there.
3. Frontend application
The actual editor/runtime lives under web/. web/js/action_director.js is the main application controller, with web/yedp_manual.html as the UI shell and web/css/style.css for layout. This is the interactive viewport, toolbar, gizmos, sequencer, and bake controls the user manipulates inside ComfyUI.
4. 3D engine and asset pipeline
The frontend is built on Three.js and bundles many loaders and helpers: GLTF/FBX/BVH/PLY/HDR loading, orbit and transform controls, skeleton utilities, mesh BVH acceleration, and a GPU path tracer. This means the viewport can ingest multiple DCC/export formats and render them with either raster or physically based methods.
5. Character subsystem
A core subsystem is the rigged character manager. It is optimized for Mixamo-compatible skeletons, with retargeting/synonym handling for variant rigs. The scene supports multiple characters, per-character animation sequences, crossfading, gendered depth mesh swapping, and prop attachments that ride along with the rig.
6. Animation and timeline data flow
Character motion comes from imported animation files and is organized into sequenced slots in the UI. The runtime keeps track of playback, scrubbing, and keyframed state so the same scene can be reproduced later from the saved workflow or from an external scene JSON file.
7. Facial mocap subsystem
Facial capture is handled entirely in-browser using MediaPipe Tasks Vision assets bundled in web/js/ (face_landmarker.task, tasks_vision.js, WASM support files). Input can be webcam or video file, and the captured facial tracks are saved as local mocap data that can then be bound to characters in the scene.
8. Environment and background assets
Environments are loaded as 3D assets from the filesystem, including support for GLB/FBX, HDRI lighting, and point cloud / Gaussian splat-style assets via PLY-related paths and Gaussian splatting modules. The repo also includes logic for proxy shading/shadows so these environments can participate in depth and silhouette passes, not just preview rendering.
9. Rendering and bake pipeline
The main output is a multi-pass bake system that produces distinct batches for pose, depth, canny, normal, shaded, alpha, and textured views. Rendering is performed in the browser with WebGL/WebGPU-adjacent GPU tooling, and the path-traced shaded mode uses the bundled three-gpu-pathtracer plus three-mesh-bvh for acceleration.
10. Persistence and file-system conventions
The system relies on ComfyUI input folders for discovery of animations, environments, cameras, and mocap assets, with a “sync folders” behavior to hot-reload new files. Scene state can also be exported to a physical .json file, so the editor is not just ephemeral UI state; it is a serializable scene document plus external asset references.
11. Operational boundary
Almost all compute-heavy work is local: browser GPU rendering, WebAssembly decoding, and webcam/video face tracking. There is no evidence of a remote service dependency; the architecture is a self-contained ComfyUI node plus a rich local web runtime.
12. What to draw in an architecture diagram
The key boxes are: ComfyUI node wrapper (Python), browser-based 3D director app, asset folders on disk, local mocap capture, and multi-pass render output back into the workflow. The main data flow is from filesystem assets and user input into the frontend scene graph, then into baked image passes and serialized scene/workflow state.
Main entry points: nodes.py nodes.py · action_director.js Action Director · Pass.js Bake Pipeline · yedp_manual.html Manual UI
Components 17
ComfyUI Wrapper
Node registration and workflow state bridge
- __init__.py · Registers the custom node package with ComfyUI
__init__.py - nodes.py · Serializes node inputs and scene state for workflows
nodes.py
Browser Director
Client-side 3D editor and rendering runtime
- Action Director · Main browser-side scene editor and runtime
web/js/action_director.js - Manual UI · Top-level page that hosts the director UI
web/js/yedp_manual.html - Style Sheet · Visual styling for the editor and viewport
web/css/style.css - Scene State · Editable scene, sequencing, and persistent director state
- 3D Viewport · Interactive scene view with controls and gizmos
- Import Stack · GLTF, FBX, BVH, PLY, HDR and related import helpers
web/js/GLTFLoader.js - Manipulators · Orbit and transform controls for editing objects
web/js/OrbitControls.js - Rig Manager · Mixamo-oriented character setup, retargeting, and props
- Anim Timeline · Playback, scrubbing, crossfades, and clip slots
- Bake Pipeline · Produces pose, depth, canny, normal, shaded, alpha, and textured outputs
web/js/Pass.js - Path Tracer · Physically based shaded render path accelerated by mesh BVH
web/js/three-gpu-pathtracer.module.js - Mesh BVH · Spatial acceleration for ray tracing and heavy geometry queries
web/js/three-mesh-bvh.module.js
Facial Mocap
In-browser face tracking and capture assets
- Face Mocap · Browser facial tracking from webcam or video input
web/js/tasks_vision.js - Vision Assets · MediaPipe task model and WASM support files
web/js/face_landmarker.task
Asset Library
3D models and bundled runtime resources
- Bundled Models · Included rig and sample scene assets
web/Yedp_Rig.glb
Connections 18
- __init__.py → nodes.py: registers · ComfyUI exposes the node implementation through the Python wrapper
- nodes.py → Action Director: serializes state · Workflow inputs and persisted scene data are handed to the browser runtime
- Manual UI → Action Director: hosts · The HTML shell boots the application controller
- Style Sheet → Manual UI: styles · UI layout is applied to the manual shell
- Action Director → Scene State: manages · Controller owns scene composition, playback state, and persistence
- Scene State → 3D Viewport: drives · Scene state is rendered into the interactive 3D viewport
- Import Stack → Scene State: imports · External 3D assets are loaded into the scene model
- Manipulators → 3D Viewport: manipulates · User interaction tools modify the viewport scene
- Rig Manager → Scene State: binds · Characters, props, and retargeted rigs become scene entities
- Anim Timeline → Rig Manager: animates · Sequenced clips and playback control drive characters
- Anim Timeline → Scene State: persists · Timeline and playback state are saved with the scene
- Bake Pipeline → 3D Viewport: renders · Viewport can output the multi-pass bake set
- Path Tracer → Bake Pipeline: shades · Path-traced shaded mode feeds the bake system
- Mesh BVH → Path Tracer: accelerates · BVH data structures speed up ray tracing and geometry queries
- Face Mocap → Rig Manager: feeds · Captured facial tracks are bound onto characters
- Vision Assets → Face Mocap: powers · Model and WASM bundles support browser face tracking
- Bundled Models → Import Stack: supplies · Bundled GLB assets provide example rigs and scenes
- nodes.py → Bake Pipeline: returns outputs · Rendered/baked results and serialized scene state flow back into ComfyUI