# tencent-hunyuan/instantcharacter architecture

> Architecture diagram of the GitHub repository tencent-hunyuan/instantcharacter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent-hunyuan/instantcharacter
- Repository: https://github.com/tencent-hunyuan/instantcharacter

## Overview

##### Project Type and Purpose
- Project Type: A Python-based inference library / demo application for personalized image generation using a diffusion-transformer pipeline.
- Purpose: Load pre-trained diffusion and adapter models (from Hugging Face), integrate custom attention/process/resampler modules, accept a reference image + text prompt, run inference (optionally offloading data to CPU), and output a stylized image.

##### File Structure
Top-Level Directories & Files:
- app.py, infer_demo.py, offload_infer_demo.py – entry-point scripts
- pipeline.py – core pipeline class (InstantCharacterFluxPipeline)
- models/ – custom modules (attn_processor.py, norm_layer.py, resampler.py, utils.py)
- README.md – usage & architecture hints
- .gitignore, License.txt, Notice.txt – repo metadata

##### Core Components
A. User Interface Layer
1. CLI Scripts (app.py, infer_demo.py, offload_infer_demo.py)
– Accept command-line args (model paths, prompts, image paths, LoRA toggles)
2. Python API (import pipeline, call methods)

B. Model Loading & Configuration
1. HuggingFace Model Hub (external service)
– Base diffusion model (FLUX.1-dev)
– IP-Adapter & style LoRA checkpoints
2. Local Checkpoint Storage
3. Pipeline Initialization
– InstantCharacterFluxPipeline.from_pretrained()
– pipe.init_adapter()

C. Custom Modules
1. models/attn_processor.py – attention-processor customizations
2. models/norm_layer.py – normalization layers
3. models/resampler.py – diffusion resampling strategies
4. models/utils.py – shared utilities

D. Inference Engine
1. PyTorch + Accelerate runtime
2. GPU/CPU Device
– offload_infer_demo.py: memory offloading to CPU when VRAM < 22 GB
3. Image Encoders
– google/siglip-so400m-patch14-384
– facebook/dinov2-giant
4. Diffusion Transformer
– applies denoising steps (num_inference_steps)
5. LoRA Injection
– pipe.with_style_lora()

E. Data Flow
1. Input:
– Text prompt
– Reference subject image (assets/girl.jpg)
– Optional style LoRA checkpoint + trigger
2. Processing:
– Encode reference image → subject embeddings
– Denoising loop with guidance_scale
– Adapter modules integrated at attention & norm layers
3. Output:
– Generated image(s) → saved to disk (PNG)

##### Relationships & Interactions
- Scripts → instantiate Pipeline → load models from Hugging Face or local storage
- Pipeline → loads custom modules (models/) → patches attention / norm / resampler
- Pipeline → calls PyTorch‐Accelerate graph → GPU/CPU execution
- Pipeline ↔ HuggingFace Hub (for downloads)
- Scripts → present CLI/UI → save images to assets or working directory

##### Architectural Patterns & Principles
- Modular Pipeline Pattern
– Separation of concerns: model loading, adapter init, inference, style injection
- Adapter Pattern
– IP-Adapter & LoRA hooks inserted into base model
- Builder/Factory
– from_pretrained() factory method to configure pipeline
- Offloading Strategy
– Dynamic device placement (GPU vs CPU)

- A zoomed-in view of the pipeline internals, showing module overrides and adapter hooks.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "User Interface Layer"
        UI1["app.py"]:::python
        UI2["infer_demo.py"]:::python
        UI3["offload_infer_demo.py"]:::python
    end

    subgraph "Pipeline Core"
        Pipeline["pipeline.py\nInstantCharacterFluxPipeline"]:::python
    end

    subgraph "Custom Modules"
        Attn["attn_processor.py"]:::python
        Norm["norm_layer.py"]:::python
        Resample["resampler.py"]:::python
        Utils["utils.py"]:::python
    end

    subgraph "External Service"
        Hugging["HuggingFace Hub"]:::external
    end

    subgraph "Storage"
        Checkpoints["Local Checkpoint Storage"]:::storage
        Assets["assets/"]:::storage
    end

    subgraph "Documentation & Metadata"
        README["README.md"]:::storage
        Gitignore[".gitignore"]:::storage
        License["License.txt"]:::storage
        Notice["Notice.txt"]:::storage
    end

    subgraph "Compute Resources"
        Compute["GPU/CPU Device\n(PyTorch + Accelerate)"]:::compute
    end

    UI1 -->|"instantiate pipeline"| Pipeline
    UI2 -->|"instantiate pipeline"| Pipeline
    UI3 -->|"instantiate pipeline + offload"| Pipeline
    Pipeline -->|"load custom modules"| Attn
    Pipeline -->|"load custom modules"| Norm
    Pipeline -->|"load custom modules"| Resample
    Pipeline -->|"load custom modules"| Utils
    Pipeline -->|"download models"| Hugging
    Hugging -->|"checkpoint weights"| Checkpoints
    Pipeline -->|"read reference image"| Assets
    Pipeline -->|"inference execution"| Compute
    UI3 -->|"offload strategy"| Compute
    Compute -->|"save outputs (PNG)"| Assets

    click UI1 "https://github.com/tencent-hunyuan/instantcharacter/blob/main/app.py"
    click UI2 "https://github.com/tencent-hunyuan/instantcharacter/blob/main/infer_demo.py"
    click UI3 "https://github.com/tencent-hunyuan/instantcharacter/blob/main/offload_infer_demo.py"
    click Pipeline "https://github.com/tencent-hunyuan/instantcharacter/blob/main/pipeline.py"
    click Attn "https://github.com/tencent-hunyuan/instantcharacter/blob/main/models/attn_processor.py"
    click Norm "https://github.com/tencent-hunyuan/instantcharacter/blob/main/models/norm_layer.py"
    click Resample "https://github.com/tencent-hunyuan/instantcharacter/blob/main/models/resampler.py"
    click Utils "https://github.com/tencent-hunyuan/instantcharacter/blob/main/models/utils.py"
    click Assets "https://github.com/tencent-hunyuan/instantcharacter/tree/main/assets/"
    click README "https://github.com/tencent-hunyuan/instantcharacter/blob/main/README.md"
    click Gitignore "https://github.com/tencent-hunyuan/instantcharacter/blob/main/.gitignore"
    click License "https://github.com/tencent-hunyuan/instantcharacter/blob/main/License.txt"
    click Notice "https://github.com/tencent-hunyuan/instantcharacter/blob/main/Notice.txt"

    classDef python fill:#a2d2ff,stroke:#000,stroke-width:1px
    classDef external fill:#ffafcc,stroke:#000,stroke-width:1px
    classDef storage fill:#caffbf,stroke:#000,stroke-width:1px
    classDef compute fill:#bdb2ff,stroke:#000,stroke-width:1px
```
