How mokshithc/multimodal-fusion-for-detection-of-colorectal-cancer-using-clinical-bert-and-phikonv2 works
This is a Streamlit clinical decision-support app for colorectal cancer MSI/MSS prediction from patient pathology-report and H&E patch features. A clinician selects a registry patient and runs inference; frozen multimodal tensors feed cross-attention classification, after which patch attention supports a ranked gallery and microscopic heatmap, with an optional Ollama narrative. Training/evaluation and the late-fusion baseline are documented product capabilities. Source excerpts establish UI inference and model/data behavior, but omit parts of the UI execution and dataset implementation, so uncertain calls are not drawn; curation and encoder details are README-documented.
Main entry points: app.py Streamlit diagnostic app · multimodal_dataset.py Multimodal dataset · cross_attentiion_train.py Cross-attention training · cross_attention_eval.py Cross-attention evaluation
Components 18
Curation and Features
- Data curation
data - Patient registry
data/patient_registry.json - Frozen patient tensors
- Multimodal dataset
src/multimodal_dataset.py
Models and Validation
- Cross-attention model
src/architectures/cross_attention_fusion.py - Late-fusion baseline
src/architectures/late_fusion_model.py - Cross-attention training
src/cross_attentiion_train.py - Cross-attention evaluation
src/cross_attention_eval.py - Late-fusion evaluation
src/late_fusion_eval.py - Validation patient keys
results/actual_validation_keys.json - Frozen model weights
models
Clinical Inference
- Streamlit diagnostic app
src/ui/app.py
Explainability
- Patch ranking and gallery
src/generate_xai.py - Phikon patch heatmap
src/generate_pixel_xai.py
Other components
- TCGA/Kaggle data
- Clinician
- Clinical browser
- Ollama Qwen scribe
Connections 27
- TCGA/Kaggle data → Data curation: supplies data
- Data curation → Patient registry: records patients
- Data curation → Frozen patient tensors: produces tensors
- Multimodal dataset → Patient registry: reads records
- Multimodal dataset → Frozen patient tensors: loads tensors
- Clinician → Clinical browser: selects patient
- Clinical browser → Streamlit diagnostic app: submits request
- Streamlit diagnostic app → Patient registry: reads patient
- Streamlit diagnostic app → Frozen patient tensors: loads features
- Streamlit diagnostic app → Frozen model weights: loads checkpoint
- Streamlit diagnostic app → Cross-attention model: runs inference
- Cross-attention model → Streamlit diagnostic app: returns probability and attention
- Streamlit diagnostic app → Patch ranking and gallery: ranks patches
- Streamlit diagnostic app → Phikon patch heatmap: generates heatmap
- Streamlit diagnostic app → Ollama Qwen scribe: requests narrative
- Cross-attention training → Multimodal dataset: loads training data
- Cross-attention training → Cross-attention model: trains model
- Cross-attention training → Validation patient keys: writes split keys
- Cross-attention training → Frozen model weights: saves checkpoint
- Cross-attention evaluation → Multimodal dataset: loads evaluation data
- Cross-attention evaluation → Validation patient keys: reads split keys
- Cross-attention evaluation → Frozen model weights: loads checkpoint
- Cross-attention evaluation → Cross-attention model: evaluates model
- Late-fusion evaluation → Multimodal dataset: loads evaluation data
- Late-fusion evaluation → Validation patient keys: reads split keys
- Late-fusion evaluation → Frozen model weights: loads checkpoint
- Late-fusion evaluation → Late-fusion baseline: evaluates baseline