# axelallen/multimodal-bert-in-medical-image-and-text-classification architecture

> Architecture diagram of the GitHub repository axelallen/multimodal-bert-in-medical-image-and-text-classification, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/axelallen/multimodal-bert-in-medical-image-and-text-classification
- Repository: https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification

## Overview

##### Project Type and Purpose
- This is a research-oriented machine learning application focused on medical image and text classification.
- The purpose is to investigate whether multimodal embeddings (combining radiology text reports and X-ray images) can improve classification performance compared to unimodal models.
- It implements a multimodal version of BERT (MMBT) with experiments on text-only, image-only, and combined modalities, leverages PyTorch and Huggingface’s Transformer models, and includes visualization approaches (integrated gradients and attention via BertViz).

##### File Structure
- The top-level folders include:
– “MMBT/”: Contains the core model implementation (mmbt.py, image.py, mmbt_config.py, mmbt_utils.py). This is the central module that fuses text and image inputs.
– “data/”: Stores input data in various formats (CSV, JSON, image label files) plus pre-trained models. This is where the raw data – including image labels and metadata – is kept.
– “preprocess/”: Contains scripts to extract and filter labels from raw datasets (although the README notes this folder is not required to run the experiments, it still outlines the preprocessing step).
– “runs/”: Directory where Tensorboard and saved model outputs from experiments are stored.
– Notebooks (files with .ipynb extension, e.g., run_mmbt.ipynb, run_bert_text_only.ipynb): These cover the pipelines for experiments, training runs, and evaluation runs.
– Various requirement files and environment YAML files that specify dependencies and platform-specific settings.

##### README for Additional Insights
- The README details the overall experimental design and architectural decisions.
- It provides information on how the models, training parameters, and experiments (text-only, multimodal, masked text evaluation, and image-only from multimodal fine-tuning) are organized.
- It points out the dual-modality input (text tokens from radiology reports and image embeddings from X-rays) and their fusion within the MMBT model.
- It also explains that the implemented attention mechanisms (with comparisons via BertViz) and integrated gradients for explainability are key building blocks.

a. Main Components:
- Data Ingestion and Preprocessing:
- Data Sources: CSV, JSON, and image files in the “data/” directory.
- Preprocessing Scripts: Found in the “preprocess/” folder; these filter and label data.
- Model Core (MMBT):
- Text Encoder: BERT-based model (using Huggingface’s “BERT-base-uncased”).
- Image Submodel: (e.g., DenseNet-based, as referenced by the image_only experiments).
- Fusion Mechanism: The multimodal integration layer that aligns and fuses image embeddings with text token embeddings.
- Utility and Configuration: Modules like mmbt_config.py and mmbt_utils.py handle settings and helper functions.
- Experimentation & Training:
- Jupyter Notebooks: “run_mmbt.ipynb”, “run_bert_text_only.ipynb”, “run_mmbt_masked_text_eval.ipynb”, etc.
- Training Logs and Checkpoints: Stored in “runs/”.
- Hyperparameter Settings: Configurations and arguments in the notebooks and associated requirements.
- Visualization and Explainability:
- Integrated Gradients: Located in “integrated_gradients/”, to analyze model explanations.
- Attention Visualization: Implemented via BertViz (demonstrated in notebooks like “bertviz_attention.ipynb”).

b. Relationships and Interactions:
- Data flows from the “data/” directory to the preprocessing scripts, then through the model.
- The training pipeline (set up in the notebooks) loads data, calls the MMBT model (and variants) and logs results to “runs/”.
- The outputs of the MMBT model (predictions and attention scores) are visualized by downstream scripts (both integrated gradients and BertViz).
- External libraries (PyTorch, Huggingface, BertViz) interact with respective components (model training, data processing, and visualization).

c. Architectural Patterns and Design Principles:
- Modular Design: Separation into data ingestion, pre-processing, model core, experiment orchestration, and visualization modules.
- Pipeline Architecture: Clear path from raw data → preprocessing → model training → evaluation/visualization.
- Experimentation & Reproducibility: Several notebooks ensure reproducibility of experiments; results are logged in the “runs/” folder.
- Extensibility: The project is designed to allow swapping between text-only and multimodal components, with flexible configuration files.

d. Relevant Technologies and Frameworks:
- Programming Language: Python.
- Deep Learning Framework: PyTorch.
- Transformer Models: Huggingface’s BERT.
- Visualization Tools: BertViz for attention and custom integrated gradients visualization.
- Data Formats: CSV, JSON, and image files (with additional requirements to create directories for image data).

##### Instructing the Principal Software Engineer
- Provide comments or annotations on key design choices like the multimodal fusion approach and integration of external libraries.
- You capture important technical details such as framework usage (PyTorch, Huggingface Transformers) and experiment orchestration (via Jupyter notebooks).
- Remember to incorporate elements such as:
- Clear labels for each module/component

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        DS["Data Sources (CSV,JSON,Images)"]:::data
    end

    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        PP["Preprocessing Scripts"]:::processing
    end

    %% Experiment Orchestration
    subgraph "Experiment Orchestration"
        NB1["Notebook: run_mmbt.ipynb"]:::experiment
        NB2["Notebook: run_bert_text_only.ipynb"]:::experiment
        NB3["Notebook: run_mmbt_masked_text_eval.ipynb"]:::experiment
    end

    %% Model Layer
    subgraph "Model Layer"
        TE["Text Encoder (BERT)"]:::model
        IE["Image Encoder (DenseNet)"]:::model
        FM["Fusion Mechanism (MMBT Integration)"]:::model
        Config["MMBT Configuration"]:::model
        Utils["MMBT Utilities"]:::model
    end

    %% Logging & Visualization
    subgraph "Logging & Visualization"
        LOGS["Logging & Checkpoints (runs/)"]:::logging
        IG["Integrated Gradients"]:::visual
        AV["Attention Visualization (BertViz)"]:::visual
    end

    %% Data Flow
    DS --> PP
    PP --> NB1
    PP --> NB2
    PP --> NB3
    NB1 -->|"triggers"| TE
    NB1 -->|"triggers"| IE
    NB2 -->|"triggers"| TE
    NB2 -->|"triggers"| IE
    NB3 -->|"triggers"| TE
    NB3 -->|"triggers"| IE
    TE --> FM
    IE --> FM
    FM --> LOGS
    LOGS --> IG
    LOGS --> AV

    %% Additional connection from Experiment Orchestration to Model Layer
    NB1 -->|"orchestrates"| FM
    NB2 -->|"orchestrates"| FM
    NB3 -->|"orchestrates"| FM
    Config --> FM
    Utils --> FM

    %% Click Events
    click DS "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/tree/master/data/"
    click PP "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/tree/master/preprocess/"
    click FM "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/blob/master/MMBT/mmbt.py"
    click IE "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/blob/master/MMBT/image.py"
    click Config "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/blob/master/MMBT/mmbt_config.py"
    click Utils "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/blob/master/MMBT/mmbt_utils.py"
    click NB1 "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/blob/master/run_mmbt.ipynb"
    click NB2 "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/blob/master/run_bert_text_only.ipynb"
    click NB3 "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/blob/master/run_mmbt_masked_text_eval.ipynb"
    click LOGS "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/tree/master/runs/"
    click IG "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/blob/master/integrated_gradients/integrated_gradients.py"
    click AV "https://github.com/axelallen/multimodal-bert-in-medical-image-and-text-classification/blob/master/bertviz_attention.ipynb"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef processing fill:#F3E79B,stroke:#000,stroke-width:2px;
    classDef experiment fill:#FFCCCB,stroke:#000,stroke-width:2px;
    classDef model fill:#C1FFC1,stroke:#000,stroke-width:2px;
    classDef logging fill:#D3D3D3,stroke:#000,stroke-width:2px;
    classDef visual fill:#E6E6FA,stroke:#000,stroke-width:2px;
```
