# tencentarc/photomaker architecture

> Architecture diagram of the GitHub repository tencentarc/photomaker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencentarc/photomaker
- Repository: https://github.com/tencentarc/photomaker

## Overview

##### Project Type and Purpose
- PhotoMaker-V2 is a Python-based ML library and demo suite built on top of Hugging Face’s Diffusers (Stable Diffusion XL) ecosystem.
- It bundles core pipeline code (`photomaker/`), feature extraction (`feature-extractor/`), inference scripts, a Gradio UI, and example notebooks/CLI to enable rapid customization of human faces via ID embedding adapters.

##### File Structure
Top-level:
- photomaker/           ← Core pipeline implementation (models, adapters, sampling, resampler)
- feature-extractor/    ← Configuration and preprocessing logic for ID image features
- inference_scripts/    ← Scripts for specialized runs (ControlNet, T2I-Adapter, IP-Adapter)
- gradio_demo/          ← Two versions of a Gradio app to expose a web UI
- examples/             ← Sample ID image folders for inference
- predict.py            ← CLI entrypoint
- photomaker_demo*.ipynb ← Jupyter notebooks demonstrating usage
- requirements.txt, pyproject.toml, cog.yaml ← Dependency, packaging, and inference-as-service config

##### Key Architectural Insights from README
- Core pipeline is a subclass of `Diffusers`’ `StableDiffusionXLPipeline`.
- Adapters (PhotoMaker, LoRA, ControlNet, T2I, IP) are loaded and fused into the sampler.
- Multiple frontends: CLI (`predict.py`), Gradio (`gradio_demo/app.py`), notebooks.
- Automatic model download via `huggingface_hub`.

##### Main Components
A. Frontend Layer
- CLI Interface (predict.py)
- Gradio Web UI (gradio_demo/app.py, app_v2.py)
- Jupyter Notebooks (photomaker_demo.ipynb)
B. API / Orchestration Layer
- Inference scripts directory (specialized orchestration for ControlNet, IP-Adapter, T2I-Adapter)
- `photomaker.pipeline` controller module
C. Core ML Pipeline
- Feature Extractor (image preprocessing, embedding config)
- PhotoMaker Adapter Module (ID embedding stack)
- StableDiffusionXLPipeline (base model loader, scheduler, sampling)
- Adapter fusion logic (LoRA, ControlNet, IP-Adapter)
- Resampler (optional up/down sampling for resolution control)
D. Data & Models
- Input: User-provided ID images (examples/*)
- External Storage: Hugging Face Hub (model & adapter weights)
- Output: Generated images saved to disk or returned to UI
E. Infrastructure & Deployment
- Docker/Cog (cog.yaml) for hosting
- Requirements & Packaging (requirements.txt, pyproject.toml)

##### Relationships and Data Flow
- User → Frontend → API Layer
- API Layer invokes Feature Extractor → obtains embeddings
- Pipeline loads base SDXL model + PhotoMaker adapter (and optional LoRAs / ControlNet)
- Scheduler/Sampler merges adapter embeddings at runtime
- Generated image tensor → optional post-processing (resampler) → saved or streamed back
- “Input Images” → Feature Extractor
- Extracted Embeddings → Adapter Fusion → Diffusion Sampler
- Sampler → Generated Image → Frontend Output

##### Architectural Patterns & Design Principles
- Layered separation: UI → Orchestrator → ML Core → Data/Models
- Plugin/Adapter pattern: PhotoMaker, LoRA, ControlNet, T2I, IP adapters are interchangeable extensions
- Dependency injection: Base model path, adapter weights, scheduler config are pluggable at runtime
- Reuse of diffusers library core versus custom adapter logic

##### Detail Checklist
[ ] Show configuration/config files (requirements.txt, cog.yaml) feeding into deployment layer
[ ] Indicate optional branches for extended adapters and where they plug in

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Frontend Layer"
        CLI["CLI Interface<br/>(predict.py)"]:::frontend
        click CLI "https://github.com/tencentarc/photomaker/blob/main/predict.py"
        GradioV1["Gradio UI V1<br/>(app.py)"]:::frontend
        click GradioV1 "https://github.com/tencentarc/photomaker/blob/main/gradio_demo/app.py"
        GradioV2["Gradio UI V2<br/>(app_v2.py)"]:::frontend
        click GradioV2 "https://github.com/tencentarc/photomaker/blob/main/gradio_demo/app_v2.py"
        Notebook1["Notebook Demo<br/>(photomaker_demo.ipynb)"]:::frontend
        click Notebook1 "https://github.com/tencentarc/photomaker/blob/main/photomaker_demo.ipynb"
        Notebook2["Notebook Style Demo<br/>(photomaker_style_demo.ipynb)"]:::frontend
        click Notebook2 "https://github.com/tencentarc/photomaker/blob/main/photomaker_style_demo.ipynb"
    end

    subgraph "API / Orchestration Layer"
        Scripts["Inference Scripts"]:::api
        click Scripts "https://github.com/tencentarc/photomaker/blob/main/inference_scripts/inference_pmv2.py"
        click Scripts "https://github.com/tencentarc/photomaker/blob/main/inference_scripts/inference_pmv2_contronet.py"
        click Scripts "https://github.com/tencentarc/photomaker/blob/main/inference_scripts/inference_pmv2_ip_adapter.py"
        click Scripts "https://github.com/tencentarc/photomaker/blob/main/inference_scripts/inference_pmv2_t2i_adapter.py"
        Orchestrator["Pipeline Controller<br/>(pipeline.py)"]:::api
        click Orchestrator "https://github.com/tencentarc/photomaker/blob/main/photomaker/pipeline.py"
    end

    subgraph "Core ML Pipeline"
        FeatureExtractor["Feature Extractor<br/>(feature-extractor/)"]:::core
        click FeatureExtractor "https://github.com/tencentarc/photomaker/blob/main/feature-extractor/preprocessor_config.json"
        AdapterModule["PhotoMaker Adapter<br/>(model.py, model_v2.py)"]:::core
        click AdapterModule "https://github.com/tencentarc/photomaker/blob/main/photomaker/model.py"
        click AdapterModule "https://github.com/tencentarc/photomaker/blob/main/photomaker/model_v2.py"
        PipelineBase["SDXL Pipeline & Fusion<br/>(pipeline*.py)"]:::core
        click PipelineBase "https://github.com/tencentarc/photomaker/blob/main/photomaker/pipeline.py"
        click PipelineBase "https://github.com/tencentarc/photomaker/blob/main/photomaker/pipeline_controlnet.py"
        click PipelineBase "https://github.com/tencentarc/photomaker/blob/main/photomaker/pipeline_t2i_adapter.py"
        Resampler["Resampler<br/>(resampler.py)"]:::core
        click Resampler "https://github.com/tencentarc/photomaker/blob/main/photomaker/resampler.py"
    end

    subgraph "Data & Models"
        InputImages["Sample Input Images<br/>(examples/)"]:::external
        click InputImages "https://github.com/tencentarc/photomaker/tree/main/examples/lenna_woman"
        click InputImages "https://github.com/tencentarc/photomaker/tree/main/examples/newton_man"
        click InputImages "https://github.com/tencentarc/photomaker/tree/main/examples/scarletthead_woman"
        click InputImages "https://github.com/tencentarc/photomaker/tree/main/examples/yangmi_woman"
        HFHub["Hugging Face Hub"]:::external
    end

    subgraph "Infrastructure & Deployment"
        Configs["Deployment & Packaging"]:::external
        click Configs "https://github.com/tencentarc/photomaker/blob/main/cog.yaml"
        click Configs "https://github.com/tencentarc/photomaker/blob/main/requirements.txt"
        click Configs "https://github.com/tencentarc/photomaker/blob/main/pyproject.toml"
    end

    %% Data Flow
    InputImages -->|"ID Images"| FeatureExtractor
    CLI -->|invokes| Orchestrator
    GradioV1 -->|invokes| Orchestrator
    GradioV2 -->|invokes| Orchestrator
    Notebook1 -->|invokes| Orchestrator
    Notebook2 -->|invokes| Orchestrator

    Scripts -->|specialized run| Orchestrator
    Orchestrator -->|extract embeddings| FeatureExtractor
    FeatureExtractor -->|embeddings| AdapterModule
    AdapterModule -->|adapter weights| PipelineBase
    HFHub -->|model & adapter weights| PipelineBase
    PipelineBase -->|latent tensor| Resampler
    Resampler -->|processed image| Orchestrator
    Orchestrator -->|Generated Images| CLI
    Orchestrator -->|Generated Images| GradioV1
    Orchestrator -->|Generated Images| GradioV2
    Orchestrator -->|Generated Images| Notebook1
    Orchestrator -->|Generated Images| Notebook2

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef api fill:#d4edda,stroke:#333,stroke-width:1px
    classDef core fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef external fill:#e2e3e5,stroke:#333,stroke-width:1px
```
