# vpulab/ovam architecture

> Architecture diagram of the GitHub repository vpulab/ovam, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vpulab/ovam
- Repository: https://github.com/vpulab/ovam

## Overview

##### Project type and purpose
- OVAM is an open-source Python library (not a web app or microservices suite) that extends text-to-image diffusion models (Stable Diffusion) to generate open-vocabulary attention maps and perform token optimization for semantic segmentation.
- It provides:
- A “hooker” component to intercept and store cross-attention maps from any Diffusers pipeline.
- An “evaluator” callable to aggregate and return attention maps for arbitrary prompts.
- A token-optimization module to refine tokens for better mask quality.
- Example notebooks and experiment scripts showcasing use cases and benchmarks.

##### File structure
- Top-level directories:
- examples/ – Jupyter notebooks demonstrating OVAM usage.
- experiments/ – scripts and notebooks for dataset generation, ablations, and evaluations.
- ovam/ – core library split into:
­– base/  (abstractions: attention_storage, block_hooker, pipeline_hooker, locator)
­– stable_diffusion/  (model-specific hooker and DAAM adaptations)
­– stable_diffusion_sa/  (self-attention hookers)
­– optimize/  (token optimization algorithms)
­– utils/  (annotation, attention ops, device setup, plotting, random seeds, text encoding)
- Config/build files: pyproject.toml, LICENSE, README.md.

##### README for additional insights
- Usage flow: instantiate Diffusers pipeline → wrap with StableDiffusionHooker → generate image (capture attention) → call OVAM evaluator → (optionally) run token optimization.
- Dependencies: PyTorch, HuggingFace Diffusers, NumPy, Matplotlib, CRF libraries.

- User Interface / Client
– Jupyter notebooks or Python scripts in examples/ and experiments/.
- External Services & Libraries
– HuggingFace Diffusers (StableDiffusionPipeline)
– PyTorch (Tensor management & backends: CUDA / MPS)
- Core OVAM Library
– Pipeline Hooker (ovam/base/pipeline_hooker.py)
– Attention Storage (ovam/base/attention_storage.py)
– Evaluator Callable (combining base + stable_diffusion modules)
– Token Optimizer (ovam/optimize/optimize.py)
– Utilities (ovam/utils/*)
- Data Artifacts
– Captured attention maps (in-memory tensors)
– Optimized token embeddings
– Generated masks / plots
- Experiments Harness
– scripts in experiments/ that orchestrate data gen, ablations, and evaluations

1. Client → StableDiffusionPipeline.from_pretrained()
2. Pipeline → Wrapped by StableDiffusionHooker (Decorator pattern)
3. Wrapped Pipeline → Generates images + emits attention tensors → Attention Storage
4. Stored attention → fed into OVAM Evaluator → outputs attention map array
5. (Optional) Attention maps + image → Token Optimizer → new token embeddings → back into Evaluator
6. Experiments scripts orchestrate the above flows in batch mode, saving outputs to disk

c. Architectural patterns & design principles:
- Hook/Decorator Pattern: pipeline_hooker intercepts forward passes without modifying the base pipeline.
- Separation of Concerns:
– Hooking & storage (base/)
– Model-specific adapters (stable_diffusion/, stable_diffusion_sa/)
– Optimization logic (optimize/)
– Utility functions (utils/)
- Extensibility: base module enables easy adaptation to any diffusion model.

d. Key technologies & libraries (annotate near each component):
- HuggingFace Diffusers 0.20.2
- PyTorch 2.x (CUDA 11.8 / MPS)
- NumPy, Matplotlib, CRF libs for post-processing

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface / Client
    subgraph "User Interface / Client"
        direction TB
        UI1["Getting Started Notebook"]:::user
        UI2["OVAM Getting Started Colab"]:::user
        UI3["OVAM Img2Img Colab"]:::user
        UI4["Experiments Scripts & Notebooks"]:::user
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        HF["HuggingFace Diffusers"]:::external
        PT["PyTorch"]:::external
    end

    %% Core OVAM Library
    subgraph "Core OVAM Library"
        direction TB
        subgraph "Hooking & Storage"
            direction TB
            PHB["Pipeline Hooker (ovam/base/pipeline_hooker.py)"]:::core
            BHB["Block Hooker (ovam/base/block_hooker.py)"]:::core
            AS["Attention Storage (ovam/base/attention_storage.py)"]:::core
            SH["Store Hooker (ovam/base/store_hooker.py)"]:::core
            HK["Hooker Interface (ovam/base/hooker.py)"]:::core
            LC["Locator (ovam/base/locator.py)"]:::core
            DB["DAAM Block (ovam/base/daam_block.py)"]:::core
            DM["DAAM Module (ovam/base/daam_module.py)"]:::core
        end
        subgraph "Model-Specific Adapters"
            direction TB
            PHSD["SD Pipeline Hooker (ovam/stable_diffusion/pipeline_hooker.py)"]:::core
            BHSD["SD Block Hooker (ovam/stable_diffusion/block_hooker.py)"]:::core
            DBSD["SD DAAM Block (ovam/stable_diffusion/daam_block.py)"]:::core
            DMSD["SD DAAM Module (ovam/stable_diffusion/daam_module.py)"]:::core
            LCSD["SD Locator (ovam/stable_diffusion/locator.py)"]:::core
            TSD["SD Test (ovam/stable_diffusion/test.py)"]:::core
            PHSAS["Self-Att Pipeline Hooker (ovam/stable_diffusion_sa/pipeline_hooker_sa.py)"]:::core
            SBHSA["Self-Att Block Hooker (ovam/stable_diffusion_sa/self_att_block_hooker.py)"]:::core
        end
        EO["Evaluator Callable"]:::core
        TO["Token Optimizer (ovam/optimize/optimize.py)"]:::core
        subgraph "Utilities"
            direction TB
            U1["annotation.py"]:::core
            U2["attention_ops.py"]:::core
            U3["dcrf.py"]:::core
            U4["dcrf_alternative.py"]:::core
            U5["device.py"]:::core
            U6["plotting.py"]:::core
            U7["random_generator.py"]:::core
            U8["text_encoding.py"]:::core
        end
    end

    %% Data Artifacts
    subgraph "Data Artifacts"
        direction TB
        DA1["Attention Maps (in-memory tensors)"]:::data
        DA2["Optimized Token Embeddings"]:::data
        DA3["Generated Masks / Plots"]:::data
    end

    %% Documentation & Config
    subgraph "Docs & Config"
        direction TB
        DIA["diagram-OVAM.svg"]:::external
        TEA["teaser.svg"]:::external
        PYP["pyproject.toml"]:::external
        LIC["LICENSE"]:::external
        REM["README.md"]:::external
    end

    %% Relationships
    UI1 -->|uses| HF
    UI1 -->|uses| PT
    UI2 -->|uses| PHSD
    UI3 -->|uses| PHSD
    UI4 -->|orchestrates| PHB
    HF -->|passed to| PHB
    PHB -->|intercepts→forwards| AS
    AS -->|stores→feeds| EO
    EO -->|outputs| DA1
    DA1 -->|input to| TO
    TO -->|generates| DA2
    DA2 -->|feeds back to| EO
    EO -->|produces| DA3

    %% Click Events - User Interface
    click UI1 "https://github.com/vpulab/ovam/blob/main/examples/getting_started.ipynb"
    click UI2 "https://github.com/vpulab/ovam/blob/main/examples/ovam_getting_started_colab.ipynb"
    click UI3 "https://github.com/vpulab/ovam/blob/main/examples/ovam_img2img_colab.ipynb"
    click UI4 "https://github.com/vpulab/ovam/tree/main/experiments/"

    %% Click Events - Core OVAM Library
    click PHB "https://github.com/vpulab/ovam/blob/main/ovam/base/pipeline_hooker.py"
    click BHB "https://github.com/vpulab/ovam/blob/main/ovam/base/block_hooker.py"
    click AS "https://github.com/vpulab/ovam/blob/main/ovam/base/attention_storage.py"
    click SH "https://github.com/vpulab/ovam/blob/main/ovam/base/store_hooker.py"
    click HK "https://github.com/vpulab/ovam/blob/main/ovam/base/hooker.py"
    click LC "https://github.com/vpulab/ovam/blob/main/ovam/base/locator.py"
    click DB "https://github.com/vpulab/ovam/blob/main/ovam/base/daam_block.py"
    click DM "https://github.com/vpulab/ovam/blob/main/ovam/base/daam_module.py"
    click PHSD "https://github.com/vpulab/ovam/blob/main/ovam/stable_diffusion/pipeline_hooker.py"
    click BHSD "https://github.com/vpulab/ovam/blob/main/ovam/stable_diffusion/block_hooker.py"
    click DBSD "https://github.com/vpulab/ovam/blob/main/ovam/stable_diffusion/daam_block.py"
    click DMSD "https://github.com/vpulab/ovam/blob/main/ovam/stable_diffusion/daam_module.py"
    click LCSD "https://github.com/vpulab/ovam/blob/main/ovam/stable_diffusion/locator.py"
    click TSD "https://github.com/vpulab/ovam/blob/main/ovam/stable_diffusion/test.py"
    click PHSAS "https://github.com/vpulab/ovam/blob/main/ovam/stable_diffusion_sa/pipeline_hooker_sa.py"
    click SBHSA "https://github.com/vpulab/ovam/blob/main/ovam/stable_diffusion_sa/self_att_block_hooker.py"
    click TO "https://github.com/vpulab/ovam/blob/main/ovam/optimize/optimize.py"
    click U1 "https://github.com/vpulab/ovam/blob/main/ovam/utils/annotation.py"
    click U2 "https://github.com/vpulab/ovam/blob/main/ovam/utils/attention_ops.py"
    click U3 "https://github.com/vpulab/ovam/blob/main/ovam/utils/dcrf.py"
    click U4 "https://github.com/vpulab/ovam/blob/main/ovam/utils/dcrf_alternative.py"
    click U5 "https://github.com/vpulab/ovam/blob/main/ovam/utils/device.py"
    click U6 "https://github.com/vpulab/ovam/blob/main/ovam/utils/plotting.py"
    click U7 "https://github.com/vpulab/ovam/blob/main/ovam/utils/random_generator.py"
    click U8 "https://github.com/vpulab/ovam/blob/main/ovam/utils/text_encoding.py"

    %% Click Events - Docs & Config
    click DIA "https://github.com/vpulab/ovam/blob/main/docs/assets/diagram-OVAM.svg"
    click TEA "https://github.com/vpulab/ovam/blob/main/docs/assets/teaser.svg"
    click PYP "https://github.com/vpulab/ovam/blob/main/pyproject.toml"
    click LIC "https://github.com/vpulab/ovam/tree/main/LICENSE"
    click REM "https://github.com/vpulab/ovam/blob/main/README.md"

    %% Styles
    classDef core fill:#add8e6,stroke:#000,stroke-width:1px
    classDef external fill:#90ee90,stroke:#000,stroke-width:1px
    classDef user fill:#ffff99,stroke:#000,stroke-width:1px
    classDef data fill:#d3d3d3,stroke:#000,stroke-width:1px
```
