# prakatheishr/epa architecture

> Architecture diagram of the GitHub repository prakatheishr/epa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prakatheishr/epa
- Repository: https://github.com/prakatheishr/epa

## Overview

This repository builds a research pipeline for ulcerative-colitis colonoscopy analysis: clinical image/text pairs train a CLIP-style representation, which supports MES classification and image-conditioned clinical captions. Evaluation covers classification and caption metrics, with separate zero-shot and linear-probe experiments. The README documents the staged checkpoint workflow; source-index references and excerpts support the shown module relationships. Some auxiliary implementation details were not sampled, so their internal connections are omitted.

Main entry points: [`source/clip_model.py`](https://github.com/prakatheishr/epa/blob/main/source/clip_model.py) (Dual encoder), [`source/train_caption_model.py`](https://github.com/prakatheishr/epa/blob/main/source/train_caption_model.py) (Caption model training), [`source/dataset.py`](https://github.com/prakatheishr/epa/blob/main/source/dataset.py) (UC multimodal dataset), [`source/train_model.py`](https://github.com/prakatheishr/epa/blob/main/source/train_model.py) (CLIP training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data and preprocessing"]
  node_dataset["UC multimodal dataset<br/>[dataset.py]"]
  node_cleaning["Caption text cleaning<br/>[text_cleaning.py]"]
end

subgraph group_representation["Representation learning"]
  node_clip_training["CLIP training<br/>[train_model.py]"]
  node_dual_encoder["Dual encoder<br/>[clip_model.py]"]
end

subgraph group_classification["MES classification"]
  node_multimodal_training["Multimodal classifier training"]
  node_zero_shot["Zero-shot MES prediction"]
  node_linear_probe["Linear probe evaluation<br/>[linear_probe.py]"]
end

subgraph group_captioning["Caption generation and evaluation"]
  node_caption_model["Prefix captioner<br/>[caption_model.py]"]
  node_caption_training["Caption model training"]
  node_caption_inference["Caption generation"]
  node_caption_evaluation["Caption evaluation<br/>[eval_captions.py]"]
end

node_clinical_data(("Image and report data"))

node_clinical_data -->|"provides samples"| node_dataset
node_clip_training -->|"loads pairs"| node_dataset
node_clip_training -->|"trains"| node_dual_encoder
node_multimodal_training -->|"loads splits"| node_dataset
node_multimodal_training -->|"uses embeddings"| node_dual_encoder
node_clip_training -->|"provides checkpoint"| node_multimodal_training
node_zero_shot -->|"encodes prototypes"| node_dual_encoder
node_linear_probe -->|"extracts embeddings"| node_dual_encoder
node_caption_training -->|"loads caption pairs"| node_dataset
node_caption_training -->|"encodes images"| node_dual_encoder
node_caption_training -->|"trains"| node_caption_model
node_caption_training -->|"cleans captions"| node_cleaning
node_clip_training -->|"provides checkpoint"| node_caption_training
node_caption_inference -->|"encodes image"| node_dual_encoder
node_caption_inference -->|"generates with"| node_caption_model
node_caption_training -->|"provides caption checkpoint"| node_caption_inference
node_caption_evaluation -->|"loads test split"| node_dataset
node_caption_evaluation -->|"encodes images"| node_dual_encoder
node_caption_evaluation -->|"generates captions"| node_caption_model
node_caption_evaluation -->|"formats references"| node_cleaning

click node_dataset "https://github.com/prakatheishr/epa/blob/main/source/dataset.py"
click node_cleaning "https://github.com/prakatheishr/epa/blob/main/source/text_cleaning.py"
click node_clip_training "https://github.com/prakatheishr/epa/blob/main/source/train_model.py"
click node_dual_encoder "https://github.com/prakatheishr/epa/blob/main/source/clip_model.py"
click node_multimodal_training "https://github.com/prakatheishr/epa/blob/main/source/train_multimodal_model.py"
click node_zero_shot "https://github.com/prakatheishr/epa/blob/main/source/zeroshot_MES_predictor.py"
click node_linear_probe "https://github.com/prakatheishr/epa/blob/main/source/linear_probe.py"
click node_caption_model "https://github.com/prakatheishr/epa/blob/main/source/caption_model.py"
click node_caption_training "https://github.com/prakatheishr/epa/blob/main/source/train_caption_model.py"
click node_caption_inference "https://github.com/prakatheishr/epa/blob/main/source/generate_captions.py"
click node_caption_evaluation "https://github.com/prakatheishr/epa/blob/main/source/eval_captions.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset,node_cleaning toneBlue
class node_clip_training,node_dual_encoder toneAmber
class node_multimodal_training,node_zero_shot,node_linear_probe toneMint
class node_caption_model,node_caption_training,node_caption_inference,node_caption_evaluation toneRose
class node_clinical_data toneIndigo
```

## Components

### Data and preprocessing

- [UC multimodal dataset](https://github.com/prakatheishr/epa/blob/main/source/dataset.py): `source/dataset.py`
- [Caption text cleaning](https://github.com/prakatheishr/epa/blob/main/source/text_cleaning.py): `source/text_cleaning.py`

### Representation learning

- [CLIP training](https://github.com/prakatheishr/epa/blob/main/source/train_model.py): `source/train_model.py`
- [Dual encoder](https://github.com/prakatheishr/epa/blob/main/source/clip_model.py): `source/clip_model.py`

### MES classification

- [Multimodal classifier training](https://github.com/prakatheishr/epa/blob/main/source/train_multimodal_model.py): `source/train_multimodal_model.py`
- [Zero-shot MES prediction](https://github.com/prakatheishr/epa/blob/main/source/zeroshot_MES_predictor.py): `source/zeroshot_MES_predictor.py`
- [Linear probe evaluation](https://github.com/prakatheishr/epa/blob/main/source/linear_probe.py): `source/linear_probe.py`

### Caption generation and evaluation

- [Prefix captioner](https://github.com/prakatheishr/epa/blob/main/source/caption_model.py): `source/caption_model.py`
- [Caption model training](https://github.com/prakatheishr/epa/blob/main/source/train_caption_model.py): `source/train_caption_model.py`
- [Caption generation](https://github.com/prakatheishr/epa/blob/main/source/generate_captions.py): `source/generate_captions.py`
- [Caption evaluation](https://github.com/prakatheishr/epa/blob/main/source/eval_captions.py): `source/eval_captions.py`

### Other components

- **Image and report data**

## Connections

- Image and report data → UC multimodal dataset: provides samples (evidence: `README.md`)
- CLIP training → UC multimodal dataset: loads pairs (evidence: `source/train_model.py`)
- CLIP training → Dual encoder: trains (evidence: `source/train_model.py`)
- Multimodal classifier training → UC multimodal dataset: loads splits (evidence: `source/train_multimodal_model.py`)
- Multimodal classifier training → Dual encoder: uses embeddings (evidence: `source/train_multimodal_model.py`)
- CLIP training → Multimodal classifier training: provides checkpoint (evidence: `README.md`)
- Zero-shot MES prediction → Dual encoder: encodes prototypes (evidence: `source/zeroshot_MES_predictor.py`)
- Linear probe evaluation → Dual encoder: extracts embeddings (evidence: `source/linear_probe.py`)
- Caption model training → UC multimodal dataset: loads caption pairs (evidence: `source/train_caption_model.py`)
- Caption model training → Dual encoder: encodes images (evidence: `source/train_caption_model.py`)
- Caption model training → Prefix captioner: trains (evidence: `source/train_caption_model.py`)
- Caption model training → Caption text cleaning: cleans captions (evidence: `source/train_caption_model.py`)
- CLIP training → Caption model training: provides checkpoint (evidence: `README.md`)
- Caption generation → Dual encoder: encodes image (evidence: `source/generate_captions.py`)
- Caption generation → Prefix captioner: generates with (evidence: `source/generate_captions.py`)
- Caption model training → Caption generation: provides caption checkpoint (evidence: `README.md`)
- Caption evaluation → UC multimodal dataset: loads test split (evidence: `source/eval_captions.py`)
- Caption evaluation → Dual encoder: encodes images (evidence: `source/eval_captions.py`)
- Caption evaluation → Prefix captioner: generates captions (evidence: `source/eval_captions.py`)
- Caption evaluation → Caption text cleaning: formats references (evidence: `source/eval_captions.py`)
