# shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction architecture

> Architecture diagram of the GitHub repository shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction
- Repository: https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction

## Overview

This is a Streamlit medical-prediction application combining clinical breast-cancer data and chest X-rays. The main workflow collects both inputs, runs separate unimodal predictions, fuses their probabilities, and presents risk results with visual and textual explanations. The graph distinguishes the app’s orchestration from inference and explanation responsibilities. Several internal details and the Groq request are not fully shown in the excerpts, so those relationships are omitted or marked as documented/uncertain.

Main entry points: [`streamlit_app.py`](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/streamlit_app.py) (Streamlit workflow), [`multimodal_fusion.py`](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py) (Cancer prediction), [`models/cancer_model.pkl`](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/models/cancer_model.pkl) (Cancer model), [`models/pneumonia_model.pth`](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/models/pneumonia_model.pth) (Pneumonia model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_application["User application"]
  node_streamlit["Streamlit workflow<br/>[streamlit_app.py]"]
end

subgraph group_inference["Prediction and fusion"]
  node_clinical["Cancer prediction"]
  node_xray["Pneumonia prediction"]
  node_fusion["Late fusion"]
  node_preprocess["Image preprocessing"]
end

subgraph group_explainability["Explainability"]
  node_shap["Clinical feature explanation"]
  node_gradcam["Grad-CAM heatmap"]
  node_narrative["Clinical narrative"]
end

subgraph group_artifacts["Model artifacts"]
  node_cancer_model["Cancer model<br/>[cancer_model.pkl]"]
  node_pneumonia_model["Pneumonia model"]
  node_preprocessor["Clinical preprocessor<br/>[preprocessor.pkl]"]
end

node_patient(("Patient"))
node_groq["Groq API"]

node_patient -->|"enters inputs"| node_streamlit
node_streamlit -->|"predicts cancer"| node_clinical
node_streamlit -->|"predicts pneumonia"| node_xray
node_xray -->|"preprocesses image"| node_preprocess
node_clinical -->|"transforms features"| node_preprocessor
node_clinical -->|"scores features"| node_cancer_model
node_xray -->|"scores image"| node_pneumonia_model
node_streamlit -->|"combines probabilities"| node_fusion
node_streamlit -->|"requests feature explanation"| node_shap
node_streamlit -->|"requests heatmap"| node_gradcam
node_streamlit -->|"requests narrative"| node_narrative
node_narrative -.->|"generates explanation"| node_groq

click node_streamlit "https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/streamlit_app.py"
click node_clinical "https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py"
click node_xray "https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py"
click node_fusion "https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py"
click node_preprocess "https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py"
click node_cancer_model "https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/models/cancer_model.pkl"
click node_pneumonia_model "https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/models/pneumonia_model.pth"
click node_preprocessor "https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/models/preprocessor.pkl"
click node_shap "https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py"
click node_gradcam "https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py"
click node_narrative "https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_streamlit toneBlue
class node_clinical,node_xray,node_fusion,node_preprocess toneAmber
class node_shap,node_gradcam,node_narrative,node_groq toneMint
class node_cancer_model,node_pneumonia_model,node_preprocessor toneRose
class node_patient toneIndigo
```

## Components

### User application

- [Streamlit workflow](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/streamlit_app.py): `streamlit_app.py`

### Prediction and fusion

- [Cancer prediction](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py): `multimodal_fusion.py`
- [Pneumonia prediction](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py): `multimodal_fusion.py`
- [Late fusion](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py): `multimodal_fusion.py`
- [Image preprocessing](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py): `multimodal_fusion.py`

### Explainability

- [Clinical feature explanation](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py): `multimodal_fusion.py`
- [Grad-CAM heatmap](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py): `multimodal_fusion.py`
- [Clinical narrative](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/multimodal_fusion.py): `multimodal_fusion.py`

### Model artifacts

- [Cancer model](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/models/cancer_model.pkl): `models/cancer_model.pkl`
- [Pneumonia model](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/models/pneumonia_model.pth): `models/pneumonia_model.pth`
- [Clinical preprocessor](https://github.com/shahbazshaddy/explainable-multimodal-ai-for-breast-cancer-and-pneumonia-prediction/blob/main/models/preprocessor.pkl): `models/preprocessor.pkl`

### Other components

- **Patient**
- **Groq API**

## Connections

- Patient → Streamlit workflow: enters inputs (evidence: `README.md`)
- Streamlit workflow → Cancer prediction: predicts cancer (evidence: `streamlit_app.py`)
- Streamlit workflow → Pneumonia prediction: predicts pneumonia (evidence: `streamlit_app.py`)
- Pneumonia prediction → Image preprocessing: preprocesses image (evidence: `multimodal_fusion.py`)
- Cancer prediction → Clinical preprocessor: transforms features (evidence: `multimodal_fusion.py`)
- Cancer prediction → Cancer model: scores features (evidence: `multimodal_fusion.py`)
- Pneumonia prediction → Pneumonia model: scores image (evidence: `multimodal_fusion.py`)
- Streamlit workflow → Late fusion: combines probabilities (evidence: `streamlit_app.py`)
- Streamlit workflow → Clinical feature explanation: requests feature explanation (evidence: `streamlit_app.py`)
- Streamlit workflow → Grad-CAM heatmap: requests heatmap (evidence: `streamlit_app.py`)
- Streamlit workflow → Clinical narrative: requests narrative (evidence: `streamlit_app.py`)
- Clinical narrative → Groq API: generates explanation (evidence: `README.md`)
