How prakatheishr/epa works
This repository builds a research pipeline for ulcerative-colitis colonoscopy analysis: clinical image/text pairs train a CLIP-style representation, which supports MES classification and image-conditioned clinical captions. Evaluation covers classification and caption metrics, with separate zero-shot and linear-probe experiments. The README documents the staged checkpoint workflow; source-index references and excerpts support the shown module relationships. Some auxiliary implementation details were not sampled, so their internal connections are omitted.
Main entry points: clip_model.py Dual encoder · train_caption_model.py Caption model training · dataset.py UC multimodal dataset · train_model.py CLIP training
Components 12
Data and preprocessing
- UC multimodal dataset
source/dataset.py - Caption text cleaning
source/text_cleaning.py
Representation learning
- CLIP training
source/train_model.py - Dual encoder
source/clip_model.py
MES classification
- Multimodal classifier training
source/train_multimodal_model.py - Zero-shot MES prediction
source/zeroshot_MES_predictor.py - Linear probe evaluation
source/linear_probe.py
Caption generation and evaluation
- Prefix captioner
source/caption_model.py - Caption model training
source/train_caption_model.py - Caption generation
source/generate_captions.py - Caption evaluation
source/eval_captions.py
Other components
- Image and report data
Connections 20
- Image and report data → UC multimodal dataset: provides samples
README.md - CLIP training → UC multimodal dataset: loads pairs
source/train_model.py - CLIP training → Dual encoder: trains
source/train_model.py - Multimodal classifier training → UC multimodal dataset: loads splits
source/train_multimodal_model.py - Multimodal classifier training → Dual encoder: uses embeddings
source/train_multimodal_model.py - CLIP training → Multimodal classifier training: provides checkpoint
README.md - Zero-shot MES prediction → Dual encoder: encodes prototypes
source/zeroshot_MES_predictor.py - Linear probe evaluation → Dual encoder: extracts embeddings
source/linear_probe.py - Caption model training → UC multimodal dataset: loads caption pairs
source/train_caption_model.py - Caption model training → Dual encoder: encodes images
source/train_caption_model.py - Caption model training → Prefix captioner: trains
source/train_caption_model.py - Caption model training → Caption text cleaning: cleans captions
source/train_caption_model.py - CLIP training → Caption model training: provides checkpoint
README.md - Caption generation → Dual encoder: encodes image
source/generate_captions.py - Caption generation → Prefix captioner: generates with
source/generate_captions.py - Caption model training → Caption generation: provides caption checkpoint
README.md - Caption evaluation → UC multimodal dataset: loads test split
source/eval_captions.py - Caption evaluation → Dual encoder: encodes images
source/eval_captions.py - Caption evaluation → Prefix captioner: generates captions
source/eval_captions.py - Caption evaluation → Caption text cleaning: formats references
source/eval_captions.py