# mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2 architecture

> Architecture diagram of the GitHub repository mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2
- Repository: https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2

## Overview

This is a Streamlit clinical decision-support app for colorectal cancer MSI/MSS prediction from patient pathology-report and H&E patch features. A clinician selects a registry patient and runs inference; frozen multimodal tensors feed cross-attention classification, after which patch attention supports a ranked gallery and microscopic heatmap, with an optional Ollama narrative. Training/evaluation and the late-fusion baseline are documented product capabilities. Source excerpts establish UI inference and model/data behavior, but omit parts of the UI execution and dataset implementation, so uncertain calls are not drawn; curation and encoder details are README-documented.

Main entry points: [`src/ui/app.py`](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/ui/app.py) (Streamlit diagnostic app), [`src/multimodal_dataset.py`](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/multimodal_dataset.py) (Multimodal dataset), [`src/cross_attentiion_train.py`](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/cross_attentiion_train.py) (Cross-attention training), [`src/cross_attention_eval.py`](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/cross_attention_eval.py) (Cross-attention evaluation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_curation["Curation and Features"]
  node_curation["Data curation"]
  node_registry[("Patient registry")]
  node_features[("Frozen patient tensors")]
  node_dataset["Multimodal dataset"]
end

subgraph group_learning["Models and Validation"]
  node_crossmodel["Cross-attention model"]
  node_latemodel["Late-fusion baseline"]
  node_trainer["Cross-attention training"]
  node_evaluation["Cross-attention evaluation"]
  node_lateeval["Late-fusion evaluation"]
  node_valkeys[("Validation patient keys")]
  node_weights[("Frozen model weights")]
end

subgraph group_clinical["Clinical Inference"]
  node_ui["Streamlit diagnostic app<br/>[app.py]"]
end

subgraph group_explain["Explainability"]
  node_attention["Patch ranking and gallery<br/>[generate_xai.py]"]
  node_heatmap["Phikon patch heatmap"]
end

node_kaggle(("TCGA/Kaggle data"))
node_clinician(("Clinician"))
node_browser(("Clinical browser"))
node_ollama["Ollama Qwen scribe"]

node_kaggle -.->|"supplies data"| node_curation
node_curation -->|"records patients"| node_registry
node_curation -->|"produces tensors"| node_features
node_dataset -->|"reads records"| node_registry
node_dataset -->|"loads tensors"| node_features
node_clinician -->|"selects patient"| node_browser
node_browser -->|"submits request"| node_ui
node_ui -->|"reads patient"| node_registry
node_ui -->|"loads features"| node_features
node_ui -->|"loads checkpoint"| node_weights
node_ui -->|"runs inference"| node_crossmodel
node_crossmodel -->|"returns probability and attention"| node_ui
node_ui -->|"ranks patches"| node_attention
node_ui -->|"generates heatmap"| node_heatmap
node_ui -.->|"requests narrative"| node_ollama
node_trainer -->|"loads training data"| node_dataset
node_trainer -->|"trains model"| node_crossmodel
node_trainer -->|"writes split keys"| node_valkeys
node_trainer -->|"saves checkpoint"| node_weights
node_evaluation -->|"loads evaluation data"| node_dataset
node_evaluation -->|"reads split keys"| node_valkeys
node_evaluation -->|"loads checkpoint"| node_weights
node_evaluation -->|"evaluates model"| node_crossmodel
node_lateeval -->|"loads evaluation data"| node_dataset
node_lateeval -->|"reads split keys"| node_valkeys
node_lateeval -->|"loads checkpoint"| node_weights
node_lateeval -->|"evaluates baseline"| node_latemodel

click node_curation "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/tree/main/data"
click node_registry "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/data/patient_registry.json"
click node_dataset "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/multimodal_dataset.py"
click node_crossmodel "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/architectures/cross_attention_fusion.py"
click node_latemodel "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/architectures/late_fusion_model.py"
click node_trainer "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/cross_attentiion_train.py"
click node_evaluation "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/cross_attention_eval.py"
click node_lateeval "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/late_fusion_eval.py"
click node_valkeys "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/results/actual_validation_keys.json"
click node_weights "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/tree/main/models"
click node_ui "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/ui/app.py"
click node_attention "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/generate_xai.py"
click node_heatmap "https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/generate_pixel_xai.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_curation,node_registry,node_features,node_dataset,node_browser toneBlue
class node_crossmodel,node_latemodel,node_trainer,node_evaluation,node_lateeval,node_valkeys,node_weights toneAmber
class node_ui toneMint
class node_attention,node_heatmap toneRose
class node_kaggle,node_clinician,node_ollama toneIndigo
```

## Components

### Curation and Features

- [Data curation](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/tree/main/data): `data`
- [Patient registry](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/data/patient_registry.json): `data/patient_registry.json`
- **Frozen patient tensors**
- [Multimodal dataset](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/multimodal_dataset.py): `src/multimodal_dataset.py`

### Models and Validation

- [Cross-attention model](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/architectures/cross_attention_fusion.py): `src/architectures/cross_attention_fusion.py`
- [Late-fusion baseline](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/architectures/late_fusion_model.py): `src/architectures/late_fusion_model.py`
- [Cross-attention training](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/cross_attentiion_train.py): `src/cross_attentiion_train.py`
- [Cross-attention evaluation](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/cross_attention_eval.py): `src/cross_attention_eval.py`
- [Late-fusion evaluation](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/late_fusion_eval.py): `src/late_fusion_eval.py`
- [Validation patient keys](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/results/actual_validation_keys.json): `results/actual_validation_keys.json`
- [Frozen model weights](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/tree/main/models): `models`

### Clinical Inference

- [Streamlit diagnostic app](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/ui/app.py): `src/ui/app.py`

### Explainability

- [Patch ranking and gallery](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/generate_xai.py): `src/generate_xai.py`
- [Phikon patch heatmap](https://github.com/mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2/blob/main/src/generate_pixel_xai.py): `src/generate_pixel_xai.py`

### Other components

- **TCGA/Kaggle data**
- **Clinician**
- **Clinical browser**
- **Ollama Qwen scribe**

## Connections

- TCGA/Kaggle data → Data curation: supplies data
- Data curation → Patient registry: records patients
- Data curation → Frozen patient tensors: produces tensors
- Multimodal dataset → Patient registry: reads records
- Multimodal dataset → Frozen patient tensors: loads tensors
- Clinician → Clinical browser: selects patient
- Clinical browser → Streamlit diagnostic app: submits request
- Streamlit diagnostic app → Patient registry: reads patient
- Streamlit diagnostic app → Frozen patient tensors: loads features
- Streamlit diagnostic app → Frozen model weights: loads checkpoint
- Streamlit diagnostic app → Cross-attention model: runs inference
- Cross-attention model → Streamlit diagnostic app: returns probability and attention
- Streamlit diagnostic app → Patch ranking and gallery: ranks patches
- Streamlit diagnostic app → Phikon patch heatmap: generates heatmap
- Streamlit diagnostic app → Ollama Qwen scribe: requests narrative
- Cross-attention training → Multimodal dataset: loads training data
- Cross-attention training → Cross-attention model: trains model
- Cross-attention training → Validation patient keys: writes split keys
- Cross-attention training → Frozen model weights: saves checkpoint
- Cross-attention evaluation → Multimodal dataset: loads evaluation data
- Cross-attention evaluation → Validation patient keys: reads split keys
- Cross-attention evaluation → Frozen model weights: loads checkpoint
- Cross-attention evaluation → Cross-attention model: evaluates model
- Late-fusion evaluation → Multimodal dataset: loads evaluation data
- Late-fusion evaluation → Validation patient keys: reads split keys
- Late-fusion evaluation → Frozen model weights: loads checkpoint
- Late-fusion evaluation → Late-fusion baseline: evaluates baseline