# meap158/comfyui-background-replacement architecture

> Architecture diagram of the GitHub repository meap158/comfyui-background-replacement, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/meap158/comfyui-background-replacement
- Repository: https://github.com/meap158/comfyui-background-replacement

## Overview

##### Project type and purpose
- This is neither a standalone web service nor a traditional client/server system, but an “extension” or “plugin” for the ComfyUI node-graph application.
- Its purpose is to supply a set of custom nodes that implement a background-replacement pipeline: extract depth and mask from an input image, generate a new background via an SDXL-Turbo model, then composite and optionally upscale the result.

A. ComfyUI Core
- Node-graph engine & GUI
- Hosts both built-in nodes and your custom nodes
B. Custom Nodes Module (directory: custom_nodes/ComfyUI-Background-Replacement)
1. depth_estimator.py
– Loads ControlNet-LoRA “depth” model
– Outputs a single-channel depth map
2. segmenter.py
– Uses a segmentation model (e.g., SAM or Shopify code)
– Outputs an RGBA image (cropped subject + alpha mask)
3. background_replacement.py
– Coordinates depth + mask
– Feeds conditioning into a ComfyUI SDXL node (generative model)
– Outputs new background and subject crop
4. image_utils.py
– Utility functions for cropping, blurring, feathering
5. captioner.py (optional)
– Generates text prompts or captions (e.g., using BLIP)
6. upscaler.py
– Applies a super-resolution model (e.g., Real-ESRGAN)
C. Models & Assets
– control-lora-depth-rank128.safetensors
– sd_xl_turbo_1.0_fp16.safetensors
– example_images/ for local testing
D. Scripts & Config
– download_models.py / .bat → fetches models into ComfyUI\models/...
– install.py / .bat → copies files into ComfyUI\custom_nodes
– workflow_background_replacement_sdxl_turbo.json → example node-graph

1. User drags an image into ComfyUI → Input Image
2. DepthEstimator Node → Depth Map
3. Segmenter Node → Cropped RGBA + Alpha Mask
4. BackgroundReplacement Node
- Reads Depth Map & Mask
- Invokes a built-in ComfyUI SDXL node (model checkpoint: sd_xl_turbo)
→ New background image
5. ImageComposite Node (inside background_replacement)
- Overlays Cropped RGBA onto generated background
→ Composited Image
6. (Optional) Upscaler Node → High-res Output
7. (Optional) Captioner Node can generate a descriptive prompt

##### Architectural patterns & design principles
- Node-graph (dataflow) architecture: each Python file registers one or more ComfyUI nodes.
- Clear separation of concerns:
- Estimation (depth, segmentation)
- Generation (SDXL)
- Post-processing (composite, upscale)
- Reuse of utility code (image_utils.py) across multiple nodes.
- External dependency isolation via download_models.* scripts.

##### Detail focus
- Don’t just show files—show logical components and their runtime interactions.
- Flow of images (and tensors) through each node.
- That the example workflow JSON simply wires these nodes together in ComfyUI’s graph.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% ComfyUI Core
    subgraph "ComfyUI Core"
        direction TB
        NodeGraph["Node-Graph Engine"]:::core
        GUI["GUI"]:::core
    end

    %% Background Replacement Pipeline (Custom Nodes)
    subgraph "Background Replacement Pipeline" 
        direction TB
        DepthEstimator["DepthEstimator Node"]:::custom
        Segmenter["Segmenter Node"]:::custom
        BackgroundReplacement["BackgroundReplacement Node"]:::custom
        ImageComposite["ImageComposite (inside BackgroundReplacement)"]:::custom
        Upscaler["Upscaler Node (optional)"]:::custom
        Captioner["Captioner Node (optional)"]:::custom
    end

    %% Models & Assets
    subgraph "Models & Assets"
        direction TB
        DepthModel["control-lora-depth-rank128.safetensors"]:::model
        SDXLModel["sd_xl_turbo_1.0_fp16.safetensors"]:::model
        ExampleImages["example_images/"]:::model
    end

    %% Scripts & Config
    subgraph "Scripts & Config"
        direction TB
        DownloadModelsPy["download_models.py"]:::config
        DownloadModelsBat["download_models.bat"]:::config
        InstallPy["install.py"]:::config
        InstallBat["install.bat"]:::config
        WorkflowJSON["workflow_background_replacement_sdxl_turbo.json"]:::config
        README["README.md"]:::config
        LICENSE["LICENSE"]:::config
        Requirements["requirements.txt"]:::config
    end

    %% Utilities Module
    ImageUtils["image_utils.py"]:::custom

    %% Data Flow
    User["User Input Image"] -->|drags into| NodeGraph
    NodeGraph -->|"feeds image"| DepthEstimator
    DepthEstimator -->|produces depth_map.png| Segmenter
    Segmenter -->|produces rgba_crop + alpha_mask.png| BackgroundReplacement
    BackgroundReplacement -->|"invokes ComfyUI SDXL"| SDXLModel
    BackgroundReplacement -->|generates new background| ImageComposite
    Segmenter --> ImageComposite
    ImageComposite -->|composited image| Upscaler
    Upscaler -->|high-res output| GUI
    BackgroundReplacement --> Captioner
    Captioner -->|caption text| GUI

    %% Hosting Relationship
    NodeGraph -->|"hosts custom nodes"| DepthEstimator
    NodeGraph --> Segmenter
    NodeGraph --> BackgroundReplacement
    NodeGraph --> Upscaler
    NodeGraph --> Captioner

    %% Model Loading
    DepthModel --> DepthEstimator
    SDXLModel --> BackgroundReplacement

    %% Utilities Usage
    ImageUtils --> DepthEstimator
    ImageUtils --> Segmenter
    ImageUtils --> BackgroundReplacement
    ImageUtils --> ImageComposite
    ImageUtils --> Upscaler

    %% Scripts & Config Links
    DownloadModelsPy --> DepthModel
    DownloadModelsBat --> DepthModel
    InstallPy -->|"installs to custom_nodes"| DepthEstimator
    InstallBat --> DepthEstimator
    WorkflowJSON -->|"example graph config"| NodeGraph

    %% Example Images for Testing
    ExampleImages --> DepthEstimator
    ExampleImages --> Segmenter

    %% Click Events
    click DepthEstimator "https://github.com/meap158/comfyui-background-replacement/blob/main/depth_estimator.py"
    click Segmenter "https://github.com/meap158/comfyui-background-replacement/blob/main/segmenter.py"
    click BackgroundReplacement "https://github.com/meap158/comfyui-background-replacement/blob/main/background_replacement.py"
    click ImageUtils "https://github.com/meap158/comfyui-background-replacement/blob/main/image_utils.py"
    click Captioner "https://github.com/meap158/comfyui-background-replacement/blob/main/captioner.py"
    click Upscaler "https://github.com/meap158/comfyui-background-replacement/blob/main/upscaler.py"
    click DownloadModelsPy "https://github.com/meap158/comfyui-background-replacement/blob/main/download_models.py"
    click DownloadModelsBat "https://github.com/meap158/comfyui-background-replacement/blob/main/download_models.bat"
    click InstallPy "https://github.com/meap158/comfyui-background-replacement/blob/main/install.py"
    click InstallBat "https://github.com/meap158/comfyui-background-replacement/blob/main/install.bat"
    click WorkflowJSON "https://github.com/meap158/comfyui-background-replacement/blob/main/workflow_background_replacement_sdxl_turbo.json"
    click ExampleImages "https://github.com/meap158/comfyui-background-replacement/tree/main/example_images/"
    click README "https://github.com/meap158/comfyui-background-replacement/blob/main/README.md"
    click LICENSE "https://github.com/meap158/comfyui-background-replacement/tree/main/LICENSE"
    click Requirements "https://github.com/meap158/comfyui-background-replacement/blob/main/requirements.txt"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#000
    classDef custom fill:#90EE90,stroke:#000
    classDef model fill:#FFA500,stroke:#000,shape:cylinder
    classDef config fill:#D3D3D3,stroke:#000
```
