# sudo-ai-3d/zero123plus architecture

> Architecture diagram of the GitHub repository sudo-ai-3d/zero123plus, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sudo-ai-3d/zero123plus
- Repository: https://github.com/sudo-ai-3d/zero123plus

## Overview

##### Project type and purpose
- This is an open-source AI inference library and demo suite centered on a custom “Zero123++” diffusion pipeline.
- It provides both programmatic entry points (Python scripts & example notebooks) and interactive UIs (Streamlit in app.py, Gradio in gradio_app.py), plus deployment config (Dockerfile, cog.yaml).

##### File structure
- Top‐level scripts:
- app.py (Streamlit demo)
- gradio_app.py (Gradio demo)
- predict.py, download_checkpoints.py (CLI entrypoints for inference and weight download)
- examples/… (usage snippets for different ControlNets and postprocessing)
- Core library code:
- diffusers-support/pipeline.py (custom DiffusionPipeline subclass)
- util/download_weights.py (helper to fetch model checkpoints)
- Deployment/config:
- Dockerfile, .dockerignore, cog.yaml (Replicate), .gitpod.yml (Cloud IDE)
- Resources:
- resources/… (teaser images, sample data placeholders)

##### Key insights from the README
- The custom pipeline plugs into Hugging Face Diffusers and Transformers.
- Supports multiple “ControlNet” variants for depth and normal generation.
- Example code shows how the pipeline is loaded from pretrained HF checkpoints, conditioned, and run on GPU.
- Interactive demos run inside Docker or cloud IDE via Streamlit/Gradio.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% UI / Entry Points (Green)
    subgraph "UI / Entry Points"
        classDef ui fill:#b2f2bb,stroke:#2f9e44,color:#000
        Streamlit(["Streamlit UI\n(app.py)"]):::ui
        Gradio(["Gradio UI\n(gradio_app.py)"]):::ui
        InferenceCLI(["Inference CLI\n(predict.py)"]):::ui
        DownloaderCLI(["Checkpoint Downloader\n(download_checkpoints.py)"]):::ui
        subgraph "Example Scripts"
            classDef examples fill:#b2f2bb,stroke:#2f9e44,color:#000
            DepthDemo(["Depth ControlNet Demo\n(examples/depth_controlnet.py)"]):::examples
            Img3DDemo(["Image-to-3D MV Demo\n(examples/img_to_mv.py)"]):::examples
            NormalDemo(["Normal Map Demo\n(examples/normal_gen.py)"]):::examples
            Text2Img(["Text-to-Image Demo\n(examples/text_to_img.py)"]):::examples
            MattingPP(["Matting Post-Processing\n(examples/matting_postprocess.py)"]):::examples
        end
    end

    %% Core Inference Engine (Blue)
    subgraph "Core Inference Engine"
        classDef core fill:#bbdefb,stroke:#1c7ed6,color:#000
        Pipeline(["CustomPipeline\n(diffusers-support/pipeline.py)"]):::core
        Libraries(["PyTorch + Diffusers + Transformers"]):::core
        subgraph "ControlNet Modules"
            classDef control fill:#bbdefb,stroke:#1c7ed6,color:#000
            DepthCN(["Depth ControlNet"]):::control
            NormalCN(["Normal ControlNet"]):::control
        end
        WeightsUtil(["Weights Utility\n(util/download_weights.py)"]):::core
    end

    %% Model Weights Store (Orange)
    subgraph "Model Weights Store"
        classDef external fill:#ffe8a1,stroke:#f59f00,color:#000
        LocalCache(["Local Cache\n(cylinder)"]):::external
        RemoteHF(["Hugging Face Endpoints\n(cylinder)"]):::external
    end

    %% Post-Processing Utilities (Blue)
    MattingLib(["rembg Background Removal"]):::core

    %% Deployment & CI/CD (Gray)
    subgraph "Deployment & CI/CD"
        classDef infra fill:#e9ecef,stroke:#868e96,color:#000
        DockerfileNode(["Dockerfile"]):::infra
        Dockerignore([".dockerignore"]):::infra
        CogConfig(["cog.yaml"]):::infra
        GitpodConfig([".gitpod.yml"]):::infra
    end

    %% Data Flow
    Streamlit -->|"calls"| Pipeline
    Gradio -->|"calls"| Pipeline
    InferenceCLI -->|"calls"| Pipeline
    DownloaderCLI -->|"invokes"| WeightsUtil
    DepthDemo -->|"imports"| Pipeline
    Img3DDemo -->|"imports"| Pipeline
    NormalDemo -->|"imports"| Pipeline
    Text2Img -->|"imports"| Pipeline
    MattingPP -->|"uses"| MattingLib

    Pipeline -->|"checks weights"| LocalCache
    LocalCache -- missing --> WeightsUtil
    WeightsUtil -->|"downloads from"| RemoteHF
    RemoteHF --> LocalCache

    Pipeline -->|"loads libs"| Libraries
    Pipeline -->|"injects conditioning"| DepthCN
    Pipeline -->|"injects conditioning"| NormalCN
    Pipeline -->|"runs inference on GPU"| Libraries
    Libraries --> Pipeline

    Pipeline -->|"returns images"| Streamlit
    Pipeline -->|"returns images"| Gradio
    Pipeline -->|"returns results"| InferenceCLI
    MattingLib -->|"post-process images"| Streamlit
    MattingLib -->|"post-process images"| Gradio

    %% Deployment Relations
    DockerfileNode -.-> Streamlit
    DockerfileNode -.-> Gradio
    CogConfig -.-> DockerfileNode
    GitpodConfig -.-> DockerfileNode

    %% Click Events
    click Streamlit "https://github.com/sudo-ai-3d/zero123plus/blob/main/app.py"
    click Gradio "https://github.com/sudo-ai-3d/zero123plus/blob/main/gradio_app.py"
    click InferenceCLI "https://github.com/sudo-ai-3d/zero123plus/blob/main/predict.py"
    click DownloaderCLI "https://github.com/sudo-ai-3d/zero123plus/blob/main/download_checkpoints.py"
    click DepthDemo "https://github.com/sudo-ai-3d/zero123plus/blob/main/examples/depth_controlnet.py"
    click Img3DDemo "https://github.com/sudo-ai-3d/zero123plus/blob/main/examples/img_to_mv.py"
    click NormalDemo "https://github.com/sudo-ai-3d/zero123plus/blob/main/examples/normal_gen.py"
    click Text2Img "https://github.com/sudo-ai-3d/zero123plus/blob/main/examples/text_to_img.py"
    click MattingPP "https://github.com/sudo-ai-3d/zero123plus/blob/main/examples/matting_postprocess.py"
    click Pipeline "https://github.com/sudo-ai-3d/zero123plus/blob/main/diffusers-support/pipeline.py"
    click WeightsUtil "https://github.com/sudo-ai-3d/zero123plus/blob/main/util/download_weights.py"
    click DockerfileNode "https://github.com/sudo-ai-3d/zero123plus/tree/main/Dockerfile"
    click Dockerignore "https://github.com/sudo-ai-3d/zero123plus/blob/main/.dockerignore"
    click CogConfig "https://github.com/sudo-ai-3d/zero123plus/blob/main/cog.yaml"
    click GitpodConfig "https://github.com/sudo-ai-3d/zero123plus/blob/main/.gitpod.yml"

    %% Styles
    class Streamlit,Gradio,InferenceCLI,DownloaderCLI ui
    class DepthDemo,Img3DDemo,NormalDemo,Text2Img,MattingPP examples
    class Pipeline,Libraries,DepthCN,NormalCN,WeightsUtil core
    class LocalCache,RemoteHF external
    class DockerfileNode,Dockerignore,CogConfig,GitpodConfig infra
```
