How thailearncoding/coco_caption_classification works
COCO Caption Classification is a multimodal image–caption matching project. Its user-facing workflow accepts an image and candidate captions, loads a zero-shot CLIP or few-shot probe, predicts the best caption, and can display image-attention maps. The broader workflow prepares few-shot data, trains probes, and evaluates accuracy/F1 and inference time. The README documents these capabilities; sampled source confirms the demo and core runtime calls, while some training and reporting links are documented rather than fully inspected.
Main entry points: notebook_utils.py Probe training · streamlit_app.py Streamlit demo · data_loader.py Caption dataset · model_arch.py Residual probes
Components 15
Data preparation
- COCO metadata
coco_subset_images/coco_multimodal_subset.json - COCO images
coco_subset_images - Few-shot splits
src/data_loader.py - Caption dataset
src/data_loader.py
Modeling and training
- CLIP encoders
src/model_arch.py - Residual probes
src/model_arch.py - Probe training
src/notebook_utils.py - Training config
settings/config.yaml - Probe checkpoints
models
Evaluation and analysis
- Metrics evaluation
src/utils.py - Training reports
src/notebook_utils.py - Results dashboard
docs/index.html
Interactive demo
- Streamlit demo
streamlit_app.py - Attention maps
src/utils.py
Other components
- User
Connections 17
- User → Streamlit demo: submits inputs
streamlit_app.py - COCO metadata → Few-shot splits: provides samples
- COCO images → Caption dataset: loads images
src/data_loader.py - Few-shot splits → Caption dataset: builds splits
src/data_loader.py - Caption dataset → Probe training: supplies batches
src/notebook_utils.py - Training config → Probe training: configures training
README.md - CLIP encoders → Residual probes: provides backbone
src/model_arch.py - Probe training → Residual probes: optimizes heads
src/notebook_utils.py - Probe training → Probe checkpoints: saves weights
- Probe checkpoints → Streamlit demo: loads weights
streamlit_app.py - Streamlit demo → CLIP encoders: instantiates models
streamlit_app.py - Streamlit demo → Residual probes: instantiates models
streamlit_app.py - Streamlit demo → Attention maps: requests maps
streamlit_app.py - Streamlit demo → Metrics evaluation: predicts candidates
streamlit_app.py - Probe training → Metrics evaluation: evaluates models
src/notebook_utils.py - Probe training → Training reports: aggregates results
src/notebook_utils.py - Training reports → Results dashboard: exports plots
README.md