How shihongl1998/llm-as-a-blackbox-optimizer works
This repository provides research workflows for optimizing vision-language prompts: classification prompts are evaluated against benchmark datasets and iteratively revised with an LLM, while prompt inversion uses an input image to produce a description and iteratively generates candidate images. The tree also documents text-to-image optimization, represented by its dedicated entry point. The CLIP implementation, dataset adapters, transforms, and prompt templates support these workflows. Some entry-point and subsystem source was not sampled, so only documented capabilities and evidenced connections are shown; unsampled internal wiring is left unspecified.
Main entry points: auto_prompt_image_classification.py Classification Optimizer · auto_prompt_prompt_inversion.py Prompt Inversion · clip.py CLIP Model · datasets Benchmark Adapters
Components 21
Prompt Workflows
- Classification Optimizer
auto_prompt_image_classification.py - Image Prompt Optimizer
auto_prompt_image_optimization.py - Prompt Inversion
auto_prompt_prompt_inversion.py
Vision Language
- CLIP Model
engine/clip/clip.py - CLIP Architecture
engine/clip/model.py - Partial Encoders
engine/clip/partial_model.py - Text Tokenizer
engine/clip/simple_tokenizer.py - Image Transforms
engine/transforms/default.py
Datasets and Inputs
- Dataset Registry
engine/datasets/__init__.py - Benchmark Adapters
engine/datasets - Benchmark Splits
engine/datasets/benchmark.py
Prompt Resources
- Prompt Templates
init_templates.py - Prompt Pool
prompt_pool.py - LAION Template Loader
load_laioncoco_templates.py - Template Initialization
init_templates.py
Other components
- Researcher
- OpenAI Chat
- GPT-4 Vision
- DALL·E
- Dataset Files
- Input Image
Connections 23
- Researcher → Classification Optimizer: runs
- Researcher → Image Prompt Optimizer: runs
- Researcher → Prompt Inversion: runs
- Classification Optimizer → Dataset Registry: loads datasets
- Dataset Registry → Benchmark Adapters: selects adapter
- Benchmark Adapters → Benchmark Splits: uses splits
- Benchmark Adapters → Dataset Files: reads images
- Classification Optimizer → CLIP Model: evaluates prompts
- CLIP Model → CLIP Architecture: builds model
- CLIP Model → Text Tokenizer: tokenizes text
- Classification Optimizer → Image Transforms: prepares images
- Classification Optimizer → Template Initialization: loads templates
- Template Initialization → Prompt Templates: selects prompts
- Template Initialization → Prompt Pool: uses pool
- LAION Template Loader → Prompt Pool: writes samples
- Classification Optimizer → OpenAI Chat: requests revisions
- Image Prompt Optimizer → OpenAI Chat: optimizes prompts
- Prompt Inversion → Input Image: encodes image
- Prompt Inversion → GPT-4 Vision: describes image
- Prompt Inversion → DALL·E: generates candidates
- Prompt Inversion → GPT-4 Vision: compares candidates
- Prompt Inversion → DALL·E: regenerates image
- Partial Encoders → CLIP Architecture: splits encoders