# shihongl1998/llm-as-a-blackbox-optimizer architecture

> Architecture diagram of the GitHub repository shihongl1998/llm-as-a-blackbox-optimizer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shihongl1998/llm-as-a-blackbox-optimizer
- Repository: https://github.com/shihongl1998/llm-as-a-blackbox-optimizer

## Overview

This repository provides research workflows for optimizing vision-language prompts: classification prompts are evaluated against benchmark datasets and iteratively revised with an LLM, while prompt inversion uses an input image to produce a description and iteratively generates candidate images. The tree also documents text-to-image optimization, represented by its dedicated entry point. The CLIP implementation, dataset adapters, transforms, and prompt templates support these workflows. Some entry-point and subsystem source was not sampled, so only documented capabilities and evidenced connections are shown; unsampled internal wiring is left unspecified.

Main entry points: [`auto_prompt_image_classification.py`](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/auto_prompt_image_classification.py) (Classification Optimizer), [`auto_prompt_prompt_inversion.py`](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/auto_prompt_prompt_inversion.py) (Prompt Inversion), [`engine/clip/clip.py`](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/clip/clip.py) (CLIP Model), [`engine/datasets`](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/tree/main/engine/datasets) (Benchmark Adapters)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Prompt Workflows"]
  node_classification["Classification Optimizer"]
  node_imageopt["Image Prompt Optimizer"]
  node_inversion["Prompt Inversion"]
end

subgraph group_vision["Vision Language"]
  node_clipapi["CLIP Model<br/>[clip.py]"]
  node_clipcore["CLIP Architecture<br/>[model.py]"]
  node_partial["Partial Encoders<br/>[partial_model.py]"]
  node_tokenizer["Text Tokenizer"]
  node_transforms["Image Transforms<br/>[default.py]"]
end

subgraph group_data["Datasets and Inputs"]
  node_dataset_registry["Dataset Registry<br/>[__init__.py]"]
  node_benchmarks["Benchmark Adapters"]
  node_benchmarkbase["Benchmark Splits<br/>[benchmark.py]"]
end

subgraph group_templates["Prompt Resources"]
  node_templates["Prompt Templates<br/>[init_templates.py]"]
  node_promptpool["Prompt Pool<br/>[prompt_pool.py]"]
  node_laion["LAION Template Loader"]
  node_templateinit["Template Initialization<br/>[init_templates.py]"]
end

node_researcher(("Researcher"))
node_chatgpt{{"OpenAI Chat"}}
node_visionapi{{"GPT-4 Vision"}}
node_dalle{{"DALL·E"}}
node_datasetfiles[("Dataset Files")]
node_sourceimage(("Input Image"))

node_researcher -->|"runs"| node_classification
node_researcher -->|"runs"| node_imageopt
node_researcher -->|"runs"| node_inversion
node_classification -->|"loads datasets"| node_dataset_registry
node_dataset_registry -->|"selects adapter"| node_benchmarks
node_benchmarks -->|"uses splits"| node_benchmarkbase
node_benchmarks -->|"reads images"| node_datasetfiles
node_classification -->|"evaluates prompts"| node_clipapi
node_clipapi -->|"builds model"| node_clipcore
node_clipapi -->|"tokenizes text"| node_tokenizer
node_classification -->|"prepares images"| node_transforms
node_classification -->|"loads templates"| node_templateinit
node_templateinit -->|"selects prompts"| node_templates
node_templateinit -->|"uses pool"| node_promptpool
node_laion -->|"writes samples"| node_promptpool
node_classification -->|"requests revisions"| node_chatgpt
node_imageopt -->|"optimizes prompts"| node_chatgpt
node_inversion -->|"encodes image"| node_sourceimage
node_inversion -->|"describes image"| node_visionapi
node_inversion -->|"generates candidates"| node_dalle
node_inversion -->|"compares candidates"| node_visionapi
node_inversion -->|"regenerates image"| node_dalle
node_partial -->|"splits encoders"| node_clipcore

click node_classification "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/auto_prompt_image_classification.py"
click node_imageopt "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/auto_prompt_image_optimization.py"
click node_inversion "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/auto_prompt_prompt_inversion.py"
click node_clipapi "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/clip/clip.py"
click node_clipcore "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/clip/model.py"
click node_partial "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/clip/partial_model.py"
click node_tokenizer "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/clip/simple_tokenizer.py"
click node_transforms "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/transforms/default.py"
click node_dataset_registry "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/datasets/__init__.py"
click node_benchmarks "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/tree/main/engine/datasets"
click node_benchmarkbase "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/datasets/benchmark.py"
click node_templates "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/init_templates.py"
click node_promptpool "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/prompt_pool.py"
click node_laion "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/load_laioncoco_templates.py"
click node_templateinit "https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/init_templates.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_classification,node_imageopt,node_inversion toneBlue
class node_clipapi,node_clipcore,node_partial,node_tokenizer,node_transforms,node_datasetfiles toneAmber
class node_dataset_registry,node_benchmarks,node_benchmarkbase toneMint
class node_templates,node_promptpool,node_laion,node_templateinit toneRose
class node_researcher,node_chatgpt,node_visionapi,node_dalle,node_sourceimage toneIndigo
```

## Components

### Prompt Workflows

- [Classification Optimizer](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/auto_prompt_image_classification.py): `auto_prompt_image_classification.py`
- [Image Prompt Optimizer](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/auto_prompt_image_optimization.py): `auto_prompt_image_optimization.py`
- [Prompt Inversion](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/auto_prompt_prompt_inversion.py): `auto_prompt_prompt_inversion.py`

### Vision Language

- [CLIP Model](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/clip/clip.py): `engine/clip/clip.py`
- [CLIP Architecture](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/clip/model.py): `engine/clip/model.py`
- [Partial Encoders](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/clip/partial_model.py): `engine/clip/partial_model.py`
- [Text Tokenizer](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/clip/simple_tokenizer.py): `engine/clip/simple_tokenizer.py`
- [Image Transforms](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/transforms/default.py): `engine/transforms/default.py`

### Datasets and Inputs

- [Dataset Registry](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/datasets/__init__.py): `engine/datasets/__init__.py`
- [Benchmark Adapters](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/tree/main/engine/datasets): `engine/datasets`
- [Benchmark Splits](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/engine/datasets/benchmark.py): `engine/datasets/benchmark.py`

### Prompt Resources

- [Prompt Templates](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/init_templates.py): `init_templates.py`
- [Prompt Pool](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/prompt_pool.py): `prompt_pool.py`
- [LAION Template Loader](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/load_laioncoco_templates.py): `load_laioncoco_templates.py`
- [Template Initialization](https://github.com/shihongl1998/llm-as-a-blackbox-optimizer/blob/main/init_templates.py): `init_templates.py`

### Other components

- **Researcher**
- **OpenAI Chat**
- **GPT-4 Vision**
- **DALL·E**
- **Dataset Files**
- **Input Image**

## Connections

- Researcher → Classification Optimizer: runs
- Researcher → Image Prompt Optimizer: runs
- Researcher → Prompt Inversion: runs
- Classification Optimizer → Dataset Registry: loads datasets
- Dataset Registry → Benchmark Adapters: selects adapter
- Benchmark Adapters → Benchmark Splits: uses splits
- Benchmark Adapters → Dataset Files: reads images
- Classification Optimizer → CLIP Model: evaluates prompts
- CLIP Model → CLIP Architecture: builds model
- CLIP Model → Text Tokenizer: tokenizes text
- Classification Optimizer → Image Transforms: prepares images
- Classification Optimizer → Template Initialization: loads templates
- Template Initialization → Prompt Templates: selects prompts
- Template Initialization → Prompt Pool: uses pool
- LAION Template Loader → Prompt Pool: writes samples
- Classification Optimizer → OpenAI Chat: requests revisions
- Image Prompt Optimizer → OpenAI Chat: optimizes prompts
- Prompt Inversion → Input Image: encodes image
- Prompt Inversion → GPT-4 Vision: describes image
- Prompt Inversion → DALL·E: generates candidates
- Prompt Inversion → GPT-4 Vision: compares candidates
- Prompt Inversion → DALL·E: regenerates image
- Partial Encoders → CLIP Architecture: splits encoders