How shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction works
This is a Streamlit medical-prediction application combining clinical breast-cancer data and chest X-rays. The main workflow collects both inputs, runs separate unimodal predictions, fuses their probabilities, and presents risk results with visual and textual explanations. The graph distinguishes the app’s orchestration from inference and explanation responsibilities. Several internal details and the Groq request are not fully shown in the excerpts, so those relationships are omitted or marked as documented/uncertain.
Main entry points: streamlit_app.py Streamlit workflow · multimodal_fusion.py Cancer prediction · cancer_model.pkl Cancer model · pneumonia_model.pth Pneumonia model
Components 13
User application
- Streamlit workflow
streamlit_app.py
Prediction and fusion
- Cancer prediction
multimodal_fusion.py - Pneumonia prediction
multimodal_fusion.py - Late fusion
multimodal_fusion.py - Image preprocessing
multimodal_fusion.py
Explainability
- Clinical feature explanation
multimodal_fusion.py - Grad-CAM heatmap
multimodal_fusion.py - Clinical narrative
multimodal_fusion.py
Model artifacts
- Cancer model
models/cancer_model.pkl - Pneumonia model
models/pneumonia_model.pth - Clinical preprocessor
models/preprocessor.pkl
Other components
- Patient
- Groq API
Connections 12
- Patient → Streamlit workflow: enters inputs
README.md - Streamlit workflow → Cancer prediction: predicts cancer
streamlit_app.py - Streamlit workflow → Pneumonia prediction: predicts pneumonia
streamlit_app.py - Pneumonia prediction → Image preprocessing: preprocesses image
multimodal_fusion.py - Cancer prediction → Clinical preprocessor: transforms features
multimodal_fusion.py - Cancer prediction → Cancer model: scores features
multimodal_fusion.py - Pneumonia prediction → Pneumonia model: scores image
multimodal_fusion.py - Streamlit workflow → Late fusion: combines probabilities
streamlit_app.py - Streamlit workflow → Clinical feature explanation: requests feature explanation
streamlit_app.py - Streamlit workflow → Grad-CAM heatmap: requests heatmap
streamlit_app.py - Streamlit workflow → Clinical narrative: requests narrative
streamlit_app.py - Clinical narrative → Groq API: generates explanation
README.md