How vpulab/ovam works
Project type and purpose
• OVAM is an open-source Python library (not a web app or microservices suite) that extends text-to-image diffusion models (Stable Diffusion) to generate open-vocabulary attention maps and perform token optimization for semantic segmentation.
• It provides:
• A “hooker” component to intercept and store cross-attention maps from any Diffusers pipeline.
• An “evaluator” callable to aggregate and return attention maps for arbitrary prompts.
• A token-optimization module to refine tokens for better mask quality.
• Example notebooks and experiment scripts showcasing use cases and benchmarks.
File structure
• Top-level directories:
• examples/ – Jupyter notebooks demonstrating OVAM usage.
• experiments/ – scripts and notebooks for dataset generation, ablations, and evaluations.
• ovam/ – core library split into:
– base/ (abstractions: attention_storage, block_hooker, pipeline_hooker, locator)
– stable_diffusion/ (model-specific hooker and DAAM adaptations)
– stable_diffusion_sa/ (self-attention hookers)
– optimize/ (token optimization algorithms)
– utils/ (annotation, attention ops, device setup, plotting, random seeds, text encoding)
• Config/build files: pyproject.toml, LICENSE, README.md.
README for additional insights
• Usage flow: instantiate Diffusers pipeline → wrap with StableDiffusionHooker → generate image (capture attention) → call OVAM evaluator → (optionally) run token optimization.
• Dependencies: PyTorch, HuggingFace Diffusers, NumPy, Matplotlib, CRF libraries.
• User Interface / Client
– Jupyter notebooks or Python scripts in examples/ and experiments/.
• External Services & Libraries
– HuggingFace Diffusers (StableDiffusionPipeline)
– PyTorch (Tensor management & backends: CUDA / MPS)
• Core OVAM Library
– Pipeline Hooker (ovam/base/pipeline_hooker.py)
– Attention Storage (ovam/base/attention_storage.py)
– Evaluator Callable (combining base + stable_diffusion modules)
– Token Optimizer (ovam/optimize/optimize.py)
– Utilities (ovam/utils/*)
• Data Artifacts
– Captured attention maps (in-memory tensors)
– Optimized token embeddings
– Generated masks / plots
• Experiments Harness
– scripts in experiments/ that orchestrate data gen, ablations, and evaluations
1. Client → StableDiffusionPipeline.from_pretrained()
2. Pipeline → Wrapped by StableDiffusionHooker (Decorator pattern)
3. Wrapped Pipeline → Generates images + emits attention tensors → Attention Storage
4. Stored attention → fed into OVAM Evaluator → outputs attention map array
5. (Optional) Attention maps + image → Token Optimizer → new token embeddings → back into Evaluator
6. Experiments scripts orchestrate the above flows in batch mode, saving outputs to disk
c. Architectural patterns & design principles:
• Hook/Decorator Pattern: pipeline_hooker intercepts forward passes without modifying the base pipeline.
• Separation of Concerns:
– Hooking & storage (base/)
– Model-specific adapters (stable_diffusion/, stable_diffusion_sa/)
– Optimization logic (optimize/)
– Utility functions (utils/)
• Extensibility: base module enables easy adaptation to any diffusion model.
d. Key technologies & libraries (annotate near each component):
• HuggingFace Diffusers 0.20.2
• PyTorch 2.x (CUDA 11.8 / MPS)
• NumPy, Matplotlib, CRF libs for post-processing