How ecnu-icalk/bpt-vlm works
BPT-VLM tunes continuous text and visual prompts for CLIP through derivative-free optimization, then evaluates saved prompts on downstream tasks. The main path is task setup, prompt generation, fitness evaluation, optimizer update, and prompt evaluation. The README also documents shallow and deep tuning, parallel population evaluation, and a demo for reusing checkpoints. Excerpts show the deep tuning loop and parts of prompt evaluation; several shallow, encoder, and evaluation details are inferred only from imports and documented capabilities.
Main entry points: Shallow_Prompt_CLIP.py Shallow prompt model · Deep_Prompt_CLIP.py Deep prompt model · BBT_VL_Shallow.py Shallow tuning · demo.py Checkpoint demo
Components 19
Tuning and evaluation
- Shallow tuning
BBT_VL_Shallow.py - Deep tuning
BBT_VL_Deep.py - Checkpoint demo
demo.py - Shallow settings
configs/shallow_prompt.yaml - Deep settings
configs/deep_prompt.yaml
Evolution strategies
- CMA-ES
algorithm/CMA_ES.py - Other strategies
algorithm
Prompted CLIP
- Shallow prompt model
model/Shallow_Prompt_CLIP.py - Deep prompt model
model/Deep_Prompt_CLIP.py - Shallow encoders
model/shallow_encoder.py - Deep encoders
model/deep_encoder.py - Evaluation analysis
model/analysis_utils.py
Data and backbone
- CLIP loading
clip/clip.py - CLIP network
clip/model.py - Few-shot datasets
dataset/general.py - CIFAR-100 data
dataset/cifar100.py - Task datasets
dataset - Prompt checkpoint
Other components
- Researcher
Connections 28
- Researcher → Shallow tuning: starts tuning
- Researcher → Deep tuning: starts tuning
- Researcher → Checkpoint demo: runs evaluation
- Shallow settings → Shallow tuning: configures
- Deep settings → Deep tuning: configures
- Shallow tuning → Shallow prompt model: evaluates prompts
- Shallow tuning → CMA-ES: asks and updates
- Shallow tuning → Other strategies: selects strategy
- Deep tuning → Deep prompt model: evaluates prompts
- Deep tuning → CMA-ES: asks and updates
- Deep prompt model → CMA-ES: stores best prompts
- Shallow prompt model → Shallow encoders: sets context
- Deep prompt model → Deep encoders: sets context
- Shallow prompt model → CLIP loading: loads backbone
- Deep prompt model → CLIP loading: loads backbone
- CLIP loading → CLIP network: builds network
- Shallow prompt model → Few-shot datasets: loads task data
- Shallow prompt model → CIFAR-100 data: loads CIFAR-100
- Deep prompt model → Few-shot datasets: loads task data
- Shallow prompt model → Evaluation analysis: uses analysis
- Deep prompt model → Evaluation analysis: uses analysis
- Task datasets → Few-shot datasets: provides images
- Task datasets → CIFAR-100 data: provides images
- Checkpoint demo → Shallow prompt model: selects model
- Checkpoint demo → Deep prompt model: selects model
- Checkpoint demo → Prompt checkpoint: loads prompts
- Prompt checkpoint → Shallow prompt model: sets prompts
- Prompt checkpoint → Deep prompt model: sets prompts