# vectorspacelab/omnigen2 architecture

> Architecture diagram of the GitHub repository vectorspacelab/omnigen2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vectorspacelab/omnigen2
- Repository: https://github.com/vectorspacelab/omnigen2

## Overview

##### Project Type and Purpose
- OmniGen2 is an open-source Python library and demo suite for advanced multimodal generation (text‐to‐image, image editing, in-context generation, chat).
- The core resides in the `omnigen2/` package; top-level scripts (`app.py`, `inference.py`, examples, shell wrappers) provide CLI/Gradio demos.

##### File Structure
- Top-level entry points:
- `app.py`, `app_chat.py` (Gradio web demos)
- `inference.py`, `inference_chat.py` (CLI tools)
- `example_*.sh`, `example.ipynb` (usage examples)
- Package layers in `omnigen2/`:
- **models/**: transformer definition, embeddings, attention processors, param repos
- **ops/triton/**: custom Triton kernels (e.g., `layer_norm.py`)
- **pipelines/**: high-level pipelines (text-to-image, chat, editing), image preprocessing, utils
- **schedulers/**: diffusion schedulers (DPMSolver, Euler, etc.)
- **utils/**: image I/O, dynamic imports
- Supporting dirs:
- `pretrained_models/`: local model‐weight storage
- `assets/`: static images and UI branding
- `results/`: output artifacts

##### Extract Key Architecture Insights from README
- Dual decoding pathways (text vs. image) with unshared parameters
- Decoupled image tokenizer in pipelines
- Supports CPU offloading, Triton optimization, multiple scheduler choices
- Provides both programmatic (scripts, library) and UI (Gradio) access

##### Main Components & Their Relationships

A. User Interface Layer
- CLI (shell scripts, `inference.py`, `example_*.sh`)
- Web UI (Gradio via `app.py`, `app_chat.py`)
→ invokes

B. Application Layer
- Entry-point scripts parse arguments, load configs, call appropriate pipeline factory in `omnigen2.pipelines`.

C. Pipeline Layer (`omnigen2.pipelines`)
- `pipeline_omnigen2.py` (text‐to‐image, editing, in-context)
- `pipeline_omnigen2_chat.py` (multimodal chat)
- Utility modules (`pipeline_utils.py`, `image_processor.py`)
→ orchestrates model, scheduler, tokenizer, utils

D. Model Layer (`omnigen2.models`)
- Transformer core (`transformer_omnigen2.py`)
- Parameter repository (`repo.py`)
- Attention processors, embeddings, block definitions
→ uses

E. Ops & Scheduler Layer
- Custom kernels (`ops/triton/layer_norm.py`)
- Diffusion schedulers (`scheduling_dpmsolver_multistep.py`, `euler_discrete.py`)

F. Data & Storage
- `pretrained_models/` (weights)
- `assets/` (UI, brand images)
- `results/` (output images)
- External: Hugging Face model hub (weights download), GPU/CPU hardware

##### Architectural Patterns & Design Principles
- Layered architecture: UI → Application → Pipeline → Model → Ops
- Pluggable schedulers: swap scheduler modules at runtime
- Hardware‐aware ops: Triton kernels for accelerated norms
- Decoupled modalities: separate parameter sets for text and image decoding

##### Technologies & Frameworks to Annotate
- Python 3.11
- PyTorch (CUDA / CPU offload)
- Triton (custom kernels)
- Gradio (web demo)
- Hugging Face (model hub, diffusers integration pending)

##### Detail Is Key
- Logical component; don’t group everything into “library”
- Call out custom Triton ops and CPU offload as distinct optimizations
- Both interactive (Gradio) and batch (CLI scripts) usage flows

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface" 
        direction TB
        UI_WebApp["Web UI: app.py, app_chat.py"]:::ui
        UI_CLI["CLI: inference.py, inference_chat.py, example_t2i.sh, example_edit.sh, example_in_context_generation.sh, example_understanding.sh"]:::ui
        UI_Examples["Usage Examples: example.ipynb, example_images/"]:::ui
    end

    %% Application Layer
    subgraph "Application Layer"
        direction TB
        App_Scripts["Entry Points: app.py, app_chat.py, inference.py, inference_chat.py"]:::app
    end

    %% Pipeline Layer
    subgraph "Pipeline Layer"
        direction TB
        Pipeline_Text["omnigen2/pipelines/omnigen2/pipeline_omnigen2.py"]:::pipeline
        Pipeline_Chat["omnigen2/pipelines/omnigen2/pipeline_omnigen2_chat.py"]:::pipeline
        Pipeline_Utils["omnigen2/pipelines/pipeline_utils.py"]:::pipeline
        Image_Processor["omnigen2/pipelines/image_processor.py"]:::pipeline
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        Transformer_Core["omnigen2/models/transformers/transformer_omnigen2.py"]:::model
        Param_Repo["omnigen2/models/transformers/repo.py"]:::model
        Attention["omnigen2/models/attention_processor.py"]:::model
        Embeddings["omnigen2/models/embeddings.py"]:::model
        Blocks["Blocks: block_lumina2.py, components.py"]:::model
    end

    %% Ops & Scheduler Layer
    subgraph "Ops & Scheduler Layer"
        direction TB
        Triton_LayerNorm["omnigen2/ops/triton/layer_norm.py"]:::ops
        Scheduler_DPMSolver["omnigen2/schedulers/scheduling_dpmsolver_multistep.py"]:::ops
        Scheduler_Euler["omnigen2/schedulers/scheduling_flow_match_euler_discrete.py"]:::ops
    end

    %% Utilities Layer
    subgraph "Utilities"
        direction TB
        Img_Util["omnigen2/utils/img_util.py"]:::utils
        Import_Util["omnigen2/utils/import_utils.py"]:::utils
    end

    %% Data & Storage
    subgraph "Data & Storage"
        direction TB
        Weights["pretrained_models/"]:::storage
        Assets["assets/"]:::storage
        Results["results/"]:::storage
        External_HF["HuggingFace Hub"]:::external
        Hardware["GPU/CPU"]:::external
    end

    %% Connections
    UI_WebApp -->|invokes| App_Scripts
    UI_CLI -->|invokes| App_Scripts
    UI_Examples -->|demonstrates| App_Scripts

    App_Scripts -->|calls| Pipeline_Text
    App_Scripts -->|calls| Pipeline_Chat

    Pipeline_Text -->|orchestrates| Transformer_Core
    Pipeline_Text -->|orchestrates| Param_Repo
    Pipeline_Text -->|orchestrates| Attention
    Pipeline_Text -->|orchestrates| Embeddings
    Pipeline_Text -->|orchestrates| Blocks
    Pipeline_Text -->|uses| Pipeline_Utils
    Pipeline_Text -->|uses| Image_Processor
    Pipeline_Text -->|selects| Scheduler_DPMSolver
    Pipeline_Text -->|selects| Scheduler_Euler
    Pipeline_Text -->|runs kernel| Triton_LayerNorm
    Pipeline_Text -->|loads weights| Weights
    Pipeline_Text -->|saves output| Results

    Pipeline_Chat -->|orchestrates| Transformer_Core
    Pipeline_Chat -->|orchestrates| Param_Repo
    Pipeline_Chat -->|orchestrates| Attention
    Pipeline_Chat -->|orchestrates| Embeddings
    Pipeline_Chat -->|orchestrates| Blocks
    Pipeline_Chat -->|uses| Pipeline_Utils
    Pipeline_Chat -->|uses| Image_Processor
    Pipeline_Chat -->|selects| Scheduler_DPMSolver
    Pipeline_Chat -->|selects| Scheduler_Euler
    Pipeline_Chat -->|runs kernel| Triton_LayerNorm
    Pipeline_Chat -->|loads weights| Weights
    Pipeline_Chat -->|saves output| Results

    Transformer_Core -->|calls| Triton_LayerNorm
    Transformer_Core -->|calls| Scheduler_DPMSolver

    Pipeline_Utils -->|helper| Img_Util
    Pipeline_Utils -->|helper| Import_Util
    Image_Processor -->|helper| Img_Util

    Weights -->|download from| External_HF
    Pipeline_Text -->|executes on| Hardware
    Pipeline_Chat -->|executes on| Hardware

    %% Click Events
    click UI_WebApp "https://github.com/vectorspacelab/omnigen2/blob/main/app.py"
    click UI_WebApp "https://github.com/vectorspacelab/omnigen2/blob/main/app_chat.py"
    click UI_CLI "https://github.com/vectorspacelab/omnigen2/blob/main/inference.py"
    click UI_CLI "https://github.com/vectorspacelab/omnigen2/blob/main/inference_chat.py"
    click UI_CLI "https://github.com/vectorspacelab/omnigen2/blob/main/example_t2i.sh"
    click UI_CLI "https://github.com/vectorspacelab/omnigen2/blob/main/example_edit.sh"
    click UI_CLI "https://github.com/vectorspacelab/omnigen2/blob/main/example_in_context_generation.sh"
    click UI_CLI "https://github.com/vectorspacelab/omnigen2/blob/main/example_understanding.sh"
    click UI_Examples "https://github.com/vectorspacelab/omnigen2/blob/main/example.ipynb"
    click UI_Examples "https://github.com/vectorspacelab/omnigen2/tree/main/example_images/"
    click App_Scripts "https://github.com/vectorspacelab/omnigen2/blob/main/app.py"
    click App_Scripts "https://github.com/vectorspacelab/omnigen2/blob/main/app_chat.py"
    click App_Scripts "https://github.com/vectorspacelab/omnigen2/blob/main/inference.py"
    click App_Scripts "https://github.com/vectorspacelab/omnigen2/blob/main/inference_chat.py"
    click Pipeline_Text "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/pipelines/omnigen2/pipeline_omnigen2.py"
    click Pipeline_Chat "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/pipelines/omnigen2/pipeline_omnigen2_chat.py"
    click Pipeline_Utils "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/pipelines/pipeline_utils.py"
    click Image_Processor "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/pipelines/image_processor.py"
    click Transformer_Core "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/models/transformers/transformer_omnigen2.py"
    click Param_Repo "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/models/transformers/repo.py"
    click Attention "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/models/attention_processor.py"
    click Embeddings "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/models/embeddings.py"
    click Blocks "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/models/transformers/block_lumina2.py"
    click Blocks "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/models/transformers/components.py"
    click Triton_LayerNorm "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/ops/triton/layer_norm.py"
    click Scheduler_DPMSolver "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/schedulers/scheduling_dpmsolver_multistep.py"
    click Scheduler_Euler "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/schedulers/scheduling_flow_match_euler_discrete.py"
    click Img_Util "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/utils/img_util.py"
    click Import_Util "https://github.com/vectorspacelab/omnigen2/blob/main/omnigen2/utils/import_utils.py"
    click Weights "https://github.com/vectorspacelab/omnigen2/tree/main/pretrained_models/"
    click Assets "https://github.com/vectorspacelab/omnigen2/tree/main/assets/"
    click Results "https://github.com/vectorspacelab/omnigen2/tree/main/results/"

    %% Styles
    classDef ui fill:#cce5ff,stroke:#004085,color:#004085;
    classDef app fill:#d4edda,stroke:#155724,color:#155724;
    classDef pipeline fill:#fff3cd,stroke:#856404,color:#856404;
    classDef model fill:#ffe5b4,stroke:#ff8c00,color:#ff8c00;
    classDef ops fill:#f8d7da,stroke:#721c24,color:#721c24;
    classDef storage fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef utils fill:#d1ecf1,stroke:#0c5460,color:#0c5460;
    classDef external fill:#f5f5f5,stroke:#343a40,color:#343a40;
```
