How meap158/comfyui-background-replacement works
Project type and purpose
• This is neither a standalone web service nor a traditional client/server system, but an “extension” or “plugin” for the ComfyUI node-graph application.
• Its purpose is to supply a set of custom nodes that implement a background-replacement pipeline: extract depth and mask from an input image, generate a new background via an SDXL-Turbo model, then composite and optionally upscale the result.
A. ComfyUI Core
• Node-graph engine & GUI
• Hosts both built-in nodes and your custom nodes
B. Custom Nodes Module (directory: custom_nodes/ComfyUI-Background-Replacement)
1. depth_estimator.py
– Loads ControlNet-LoRA “depth” model
– Outputs a single-channel depth map
2. segmenter.py
– Uses a segmentation model (e.g., SAM or Shopify code)
– Outputs an RGBA image (cropped subject + alpha mask)
3. background_replacement.py
– Coordinates depth + mask
– Feeds conditioning into a ComfyUI SDXL node (generative model)
– Outputs new background and subject crop
4. image_utils.py
– Utility functions for cropping, blurring, feathering
5. captioner.py (optional)
– Generates text prompts or captions (e.g., using BLIP)
6. upscaler.py
– Applies a super-resolution model (e.g., Real-ESRGAN)
C. Models & Assets
– control-lora-depth-rank128.safetensors
– sd_xl_turbo_1.0_fp16.safetensors
– example_images/ for local testing
D. Scripts & Config
– download_models.py / .bat → fetches models into ComfyUI\models/...
– install.py / .bat → copies files into ComfyUI\custom_nodes
– workflow_background_replacement_sdxl_turbo.json → example node-graph
1. User drags an image into ComfyUI → Input Image
2. DepthEstimator Node → Depth Map
3. Segmenter Node → Cropped RGBA + Alpha Mask
4. BackgroundReplacement Node
• Reads Depth Map & Mask
• Invokes a built-in ComfyUI SDXL node (model checkpoint: sd_xl_turbo)
→ New background image
5. ImageComposite Node (inside background_replacement)
• Overlays Cropped RGBA onto generated background
→ Composited Image
6. (Optional) Upscaler Node → High-res Output
7. (Optional) Captioner Node can generate a descriptive prompt
Architectural patterns & design principles
• Node-graph (dataflow) architecture: each Python file registers one or more ComfyUI nodes.
• Clear separation of concerns:
• Estimation (depth, segmentation)
• Generation (SDXL)
• Post-processing (composite, upscale)
• Reuse of utility code (image_utils.py) across multiple nodes.
• External dependency isolation via download_models.* scripts.
Detail focus
• Don’t just show files—show logical components and their runtime interactions.
• Flow of images (and tensors) through each node.
• That the example workflow JSON simply wires these nodes together in ComfyUI’s graph.