# ecnu-icalk/bpt-vlm architecture

> Architecture diagram of the GitHub repository ecnu-icalk/bpt-vlm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ecnu-icalk/bpt-vlm
- Repository: https://github.com/ecnu-icalk/bpt-vlm

## Overview

BPT-VLM tunes continuous text and visual prompts for CLIP through derivative-free optimization, then evaluates saved prompts on downstream tasks. The main path is task setup, prompt generation, fitness evaluation, optimizer update, and prompt evaluation. The README also documents shallow and deep tuning, parallel population evaluation, and a demo for reusing checkpoints. Excerpts show the deep tuning loop and parts of prompt evaluation; several shallow, encoder, and evaluation details are inferred only from imports and documented capabilities.

Main entry points: [`model/Shallow_Prompt_CLIP.py`](https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/Shallow_Prompt_CLIP.py) (Shallow prompt model), [`model/Deep_Prompt_CLIP.py`](https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/Deep_Prompt_CLIP.py) (Deep prompt model), [`BBT_VL_Shallow.py`](https://github.com/ecnu-icalk/bpt-vlm/blob/master/BBT_VL_Shallow.py) (Shallow tuning), [`demo.py`](https://github.com/ecnu-icalk/bpt-vlm/blob/master/demo.py) (Checkpoint demo)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Tuning and evaluation"]
  node_tune_shallow["Shallow tuning<br/>[BBT_VL_Shallow.py]"]
  node_tune_deep["Deep tuning<br/>[BBT_VL_Deep.py]"]
  node_demo["Checkpoint demo<br/>[demo.py]"]
  node_shallow_cfg["Shallow settings"]
  node_deep_cfg["Deep settings<br/>[deep_prompt.yaml]"]
end

subgraph group_optimization["Evolution strategies"]
  node_cma["CMA-ES<br/>[CMA_ES.py]"]
  node_other_es["Other strategies"]
end

subgraph group_prompts["Prompted CLIP"]
  node_shallow_prompt["Shallow prompt model"]
  node_deep_prompt["Deep prompt model"]
  node_shallow_encoder["Shallow encoders<br/>[shallow_encoder.py]"]
  node_deep_encoder["Deep encoders<br/>[deep_encoder.py]"]
  node_analysis["Evaluation analysis<br/>[analysis_utils.py]"]
end

subgraph group_data["Data and backbone"]
  node_clip_api["CLIP loading<br/>[clip.py]"]
  node_clip_model["CLIP network<br/>[model.py]"]
  node_general_data["Few-shot datasets<br/>[general.py]"]
  node_cifar_data["CIFAR-100 data<br/>[cifar100.py]"]
  node_task_data[("Task datasets")]
  node_checkpoint["Prompt checkpoint"]
end

node_researcher(("Researcher"))

node_researcher -->|"starts tuning"| node_tune_shallow
node_researcher -->|"starts tuning"| node_tune_deep
node_researcher -->|"runs evaluation"| node_demo
node_shallow_cfg -->|"configures"| node_tune_shallow
node_deep_cfg -->|"configures"| node_tune_deep
node_tune_shallow -->|"evaluates prompts"| node_shallow_prompt
node_tune_shallow -->|"asks and updates"| node_cma
node_tune_shallow -.->|"selects strategy"| node_other_es
node_tune_deep -->|"evaluates prompts"| node_deep_prompt
node_tune_deep -->|"asks and updates"| node_cma
node_deep_prompt -->|"stores best prompts"| node_cma
node_shallow_prompt -->|"sets context"| node_shallow_encoder
node_deep_prompt -->|"sets context"| node_deep_encoder
node_shallow_prompt -->|"loads backbone"| node_clip_api
node_deep_prompt -->|"loads backbone"| node_clip_api
node_clip_api -->|"builds network"| node_clip_model
node_shallow_prompt -->|"loads task data"| node_general_data
node_shallow_prompt -->|"loads CIFAR-100"| node_cifar_data
node_deep_prompt -->|"loads task data"| node_general_data
node_shallow_prompt -->|"uses analysis"| node_analysis
node_deep_prompt -->|"uses analysis"| node_analysis
node_task_data -->|"provides images"| node_general_data
node_task_data -->|"provides images"| node_cifar_data
node_demo -->|"selects model"| node_shallow_prompt
node_demo -->|"selects model"| node_deep_prompt
node_demo -->|"loads prompts"| node_checkpoint
node_checkpoint -->|"sets prompts"| node_shallow_prompt
node_checkpoint -->|"sets prompts"| node_deep_prompt

click node_tune_shallow "https://github.com/ecnu-icalk/bpt-vlm/blob/master/BBT_VL_Shallow.py"
click node_tune_deep "https://github.com/ecnu-icalk/bpt-vlm/blob/master/BBT_VL_Deep.py"
click node_demo "https://github.com/ecnu-icalk/bpt-vlm/blob/master/demo.py"
click node_shallow_cfg "https://github.com/ecnu-icalk/bpt-vlm/blob/master/configs/shallow_prompt.yaml"
click node_deep_cfg "https://github.com/ecnu-icalk/bpt-vlm/blob/master/configs/deep_prompt.yaml"
click node_cma "https://github.com/ecnu-icalk/bpt-vlm/blob/master/algorithm/CMA_ES.py"
click node_other_es "https://github.com/ecnu-icalk/bpt-vlm/tree/master/algorithm"
click node_shallow_prompt "https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/Shallow_Prompt_CLIP.py"
click node_deep_prompt "https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/Deep_Prompt_CLIP.py"
click node_shallow_encoder "https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/shallow_encoder.py"
click node_deep_encoder "https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/deep_encoder.py"
click node_analysis "https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/analysis_utils.py"
click node_clip_api "https://github.com/ecnu-icalk/bpt-vlm/blob/master/clip/clip.py"
click node_clip_model "https://github.com/ecnu-icalk/bpt-vlm/blob/master/clip/model.py"
click node_general_data "https://github.com/ecnu-icalk/bpt-vlm/blob/master/dataset/general.py"
click node_cifar_data "https://github.com/ecnu-icalk/bpt-vlm/blob/master/dataset/cifar100.py"
click node_task_data "https://github.com/ecnu-icalk/bpt-vlm/tree/master/dataset"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tune_shallow,node_tune_deep,node_demo,node_shallow_cfg,node_deep_cfg toneBlue
class node_cma,node_other_es toneAmber
class node_shallow_prompt,node_deep_prompt,node_shallow_encoder,node_deep_encoder,node_analysis toneMint
class node_clip_api,node_clip_model,node_general_data,node_cifar_data,node_task_data,node_checkpoint toneRose
class node_researcher toneIndigo
```

## Components

### Tuning and evaluation

- [Shallow tuning](https://github.com/ecnu-icalk/bpt-vlm/blob/master/BBT_VL_Shallow.py): `BBT_VL_Shallow.py`
- [Deep tuning](https://github.com/ecnu-icalk/bpt-vlm/blob/master/BBT_VL_Deep.py): `BBT_VL_Deep.py`
- [Checkpoint demo](https://github.com/ecnu-icalk/bpt-vlm/blob/master/demo.py): `demo.py`
- [Shallow settings](https://github.com/ecnu-icalk/bpt-vlm/blob/master/configs/shallow_prompt.yaml): `configs/shallow_prompt.yaml`
- [Deep settings](https://github.com/ecnu-icalk/bpt-vlm/blob/master/configs/deep_prompt.yaml): `configs/deep_prompt.yaml`

### Evolution strategies

- [CMA-ES](https://github.com/ecnu-icalk/bpt-vlm/blob/master/algorithm/CMA_ES.py): `algorithm/CMA_ES.py`
- [Other strategies](https://github.com/ecnu-icalk/bpt-vlm/tree/master/algorithm): `algorithm`

### Prompted CLIP

- [Shallow prompt model](https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/Shallow_Prompt_CLIP.py): `model/Shallow_Prompt_CLIP.py`
- [Deep prompt model](https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/Deep_Prompt_CLIP.py): `model/Deep_Prompt_CLIP.py`
- [Shallow encoders](https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/shallow_encoder.py): `model/shallow_encoder.py`
- [Deep encoders](https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/deep_encoder.py): `model/deep_encoder.py`
- [Evaluation analysis](https://github.com/ecnu-icalk/bpt-vlm/blob/master/model/analysis_utils.py): `model/analysis_utils.py`

### Data and backbone

- [CLIP loading](https://github.com/ecnu-icalk/bpt-vlm/blob/master/clip/clip.py): `clip/clip.py`
- [CLIP network](https://github.com/ecnu-icalk/bpt-vlm/blob/master/clip/model.py): `clip/model.py`
- [Few-shot datasets](https://github.com/ecnu-icalk/bpt-vlm/blob/master/dataset/general.py): `dataset/general.py`
- [CIFAR-100 data](https://github.com/ecnu-icalk/bpt-vlm/blob/master/dataset/cifar100.py): `dataset/cifar100.py`
- [Task datasets](https://github.com/ecnu-icalk/bpt-vlm/tree/master/dataset): `dataset`
- **Prompt checkpoint**

### Other components

- **Researcher**

## Connections

- Researcher → Shallow tuning: starts tuning
- Researcher → Deep tuning: starts tuning
- Researcher → Checkpoint demo: runs evaluation
- Shallow settings → Shallow tuning: configures
- Deep settings → Deep tuning: configures
- Shallow tuning → Shallow prompt model: evaluates prompts
- Shallow tuning → CMA-ES: asks and updates
- Shallow tuning → Other strategies: selects strategy
- Deep tuning → Deep prompt model: evaluates prompts
- Deep tuning → CMA-ES: asks and updates
- Deep prompt model → CMA-ES: stores best prompts
- Shallow prompt model → Shallow encoders: sets context
- Deep prompt model → Deep encoders: sets context
- Shallow prompt model → CLIP loading: loads backbone
- Deep prompt model → CLIP loading: loads backbone
- CLIP loading → CLIP network: builds network
- Shallow prompt model → Few-shot datasets: loads task data
- Shallow prompt model → CIFAR-100 data: loads CIFAR-100
- Deep prompt model → Few-shot datasets: loads task data
- Shallow prompt model → Evaluation analysis: uses analysis
- Deep prompt model → Evaluation analysis: uses analysis
- Task datasets → Few-shot datasets: provides images
- Task datasets → CIFAR-100 data: provides images
- Checkpoint demo → Shallow prompt model: selects model
- Checkpoint demo → Deep prompt model: selects model
- Checkpoint demo → Prompt checkpoint: loads prompts
- Prompt checkpoint → Shallow prompt model: sets prompts
- Prompt checkpoint → Deep prompt model: sets prompts