# thailearncoding/coco_caption_classification architecture

> Architecture diagram of the GitHub repository thailearncoding/coco_caption_classification, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thailearncoding/coco_caption_classification
- Repository: https://github.com/thailearncoding/coco_caption_classification

## Overview

COCO Caption Classification is a multimodal image–caption matching project. Its user-facing workflow accepts an image and candidate captions, loads a zero-shot CLIP or few-shot probe, predicts the best caption, and can display image-attention maps. The broader workflow prepares few-shot data, trains probes, and evaluates accuracy/F1 and inference time. The README documents these capabilities; sampled source confirms the demo and core runtime calls, while some training and reporting links are documented rather than fully inspected.

Main entry points: [`src/notebook_utils.py`](https://github.com/thailearncoding/coco_caption_classification/blob/main/src/notebook_utils.py) (Probe training), [`streamlit_app.py`](https://github.com/thailearncoding/coco_caption_classification/blob/main/streamlit_app.py) (Streamlit demo), [`src/data_loader.py`](https://github.com/thailearncoding/coco_caption_classification/blob/main/src/data_loader.py) (Caption dataset), [`src/model_arch.py`](https://github.com/thailearncoding/coco_caption_classification/blob/main/src/model_arch.py) (Residual probes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_metadata["COCO metadata"]
  node_images["COCO images"]
  node_sampling["Few-shot splits<br/>[data_loader.py]"]
  node_dataset["Caption dataset<br/>[data_loader.py]"]
end

subgraph group_models["Modeling and training"]
  node_clip["CLIP encoders<br/>[model_arch.py]"]
  node_probes["Residual probes<br/>[model_arch.py]"]
  node_training["Probe training<br/>[notebook_utils.py]"]
  node_config["Training config<br/>[config.yaml]"]
  node_checkpoints["Probe checkpoints"]
end

subgraph group_analysis["Evaluation and analysis"]
  node_evaluation["Metrics evaluation<br/>[utils.py]"]
  node_reporting["Training reports<br/>[notebook_utils.py]"]
  node_results["Results dashboard<br/>[index.html]"]
end

subgraph group_demo["Interactive demo"]
  node_demo["Streamlit demo<br/>[streamlit_app.py]"]
  node_attention["Attention maps<br/>[utils.py]"]
end

node_user(("User"))

node_user -->|"submits inputs"| node_demo
node_metadata -.->|"provides samples"| node_sampling
node_images -->|"loads images"| node_dataset
node_sampling -->|"builds splits"| node_dataset
node_dataset -->|"supplies batches"| node_training
node_config -.->|"configures training"| node_training
node_clip -->|"provides backbone"| node_probes
node_training -->|"optimizes heads"| node_probes
node_training -.->|"saves weights"| node_checkpoints
node_checkpoints -->|"loads weights"| node_demo
node_demo -->|"instantiates models"| node_clip
node_demo -->|"instantiates models"| node_probes
node_demo -->|"requests maps"| node_attention
node_demo -->|"predicts candidates"| node_evaluation
node_training -->|"evaluates models"| node_evaluation
node_training -->|"aggregates results"| node_reporting
node_reporting -.->|"exports plots"| node_results

click node_metadata "https://github.com/thailearncoding/coco_caption_classification/blob/main/coco_subset_images/coco_multimodal_subset.json"
click node_images "https://github.com/thailearncoding/coco_caption_classification/tree/main/coco_subset_images"
click node_sampling "https://github.com/thailearncoding/coco_caption_classification/blob/main/src/data_loader.py"
click node_dataset "https://github.com/thailearncoding/coco_caption_classification/blob/main/src/data_loader.py"
click node_clip "https://github.com/thailearncoding/coco_caption_classification/blob/main/src/model_arch.py"
click node_probes "https://github.com/thailearncoding/coco_caption_classification/blob/main/src/model_arch.py"
click node_training "https://github.com/thailearncoding/coco_caption_classification/blob/main/src/notebook_utils.py"
click node_config "https://github.com/thailearncoding/coco_caption_classification/blob/main/settings/config.yaml"
click node_checkpoints "https://github.com/thailearncoding/coco_caption_classification/tree/main/models"
click node_evaluation "https://github.com/thailearncoding/coco_caption_classification/blob/main/src/utils.py"
click node_reporting "https://github.com/thailearncoding/coco_caption_classification/blob/main/src/notebook_utils.py"
click node_results "https://github.com/thailearncoding/coco_caption_classification/blob/main/docs/index.html"
click node_demo "https://github.com/thailearncoding/coco_caption_classification/blob/main/streamlit_app.py"
click node_attention "https://github.com/thailearncoding/coco_caption_classification/blob/main/src/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_metadata,node_images,node_sampling,node_dataset,node_user toneBlue
class node_clip,node_probes,node_training,node_config,node_checkpoints toneAmber
class node_evaluation,node_reporting,node_results toneMint
class node_demo,node_attention toneRose
```

## Components

### Data preparation

- [COCO metadata](https://github.com/thailearncoding/coco_caption_classification/blob/main/coco_subset_images/coco_multimodal_subset.json): `coco_subset_images/coco_multimodal_subset.json`
- [COCO images](https://github.com/thailearncoding/coco_caption_classification/tree/main/coco_subset_images): `coco_subset_images`
- [Few-shot splits](https://github.com/thailearncoding/coco_caption_classification/blob/main/src/data_loader.py): `src/data_loader.py`
- [Caption dataset](https://github.com/thailearncoding/coco_caption_classification/blob/main/src/data_loader.py): `src/data_loader.py`

### Modeling and training

- [CLIP encoders](https://github.com/thailearncoding/coco_caption_classification/blob/main/src/model_arch.py): `src/model_arch.py`
- [Residual probes](https://github.com/thailearncoding/coco_caption_classification/blob/main/src/model_arch.py): `src/model_arch.py`
- [Probe training](https://github.com/thailearncoding/coco_caption_classification/blob/main/src/notebook_utils.py): `src/notebook_utils.py`
- [Training config](https://github.com/thailearncoding/coco_caption_classification/blob/main/settings/config.yaml): `settings/config.yaml`
- [Probe checkpoints](https://github.com/thailearncoding/coco_caption_classification/tree/main/models): `models`

### Evaluation and analysis

- [Metrics evaluation](https://github.com/thailearncoding/coco_caption_classification/blob/main/src/utils.py): `src/utils.py`
- [Training reports](https://github.com/thailearncoding/coco_caption_classification/blob/main/src/notebook_utils.py): `src/notebook_utils.py`
- [Results dashboard](https://github.com/thailearncoding/coco_caption_classification/blob/main/docs/index.html): `docs/index.html`

### Interactive demo

- [Streamlit demo](https://github.com/thailearncoding/coco_caption_classification/blob/main/streamlit_app.py): `streamlit_app.py`
- [Attention maps](https://github.com/thailearncoding/coco_caption_classification/blob/main/src/utils.py): `src/utils.py`

### Other components

- **User**

## Connections

- User → Streamlit demo: submits inputs (evidence: `streamlit_app.py`)
- COCO metadata → Few-shot splits: provides samples
- COCO images → Caption dataset: loads images (evidence: `src/data_loader.py`)
- Few-shot splits → Caption dataset: builds splits (evidence: `src/data_loader.py`)
- Caption dataset → Probe training: supplies batches (evidence: `src/notebook_utils.py`)
- Training config → Probe training: configures training (evidence: `README.md`)
- CLIP encoders → Residual probes: provides backbone (evidence: `src/model_arch.py`)
- Probe training → Residual probes: optimizes heads (evidence: `src/notebook_utils.py`)
- Probe training → Probe checkpoints: saves weights
- Probe checkpoints → Streamlit demo: loads weights (evidence: `streamlit_app.py`)
- Streamlit demo → CLIP encoders: instantiates models (evidence: `streamlit_app.py`)
- Streamlit demo → Residual probes: instantiates models (evidence: `streamlit_app.py`)
- Streamlit demo → Attention maps: requests maps (evidence: `streamlit_app.py`)
- Streamlit demo → Metrics evaluation: predicts candidates (evidence: `streamlit_app.py`)
- Probe training → Metrics evaluation: evaluates models (evidence: `src/notebook_utils.py`)
- Probe training → Training reports: aggregates results (evidence: `src/notebook_utils.py`)
- Training reports → Results dashboard: exports plots (evidence: `README.md`)
