How axelallen/multimodal-bert-in-medical-image-and-text-classification works
Project Type and Purpose
• This is a research-oriented machine learning application focused on medical image and text classification.
• The purpose is to investigate whether multimodal embeddings (combining radiology text reports and X-ray images) can improve classification performance compared to unimodal models.
• It implements a multimodal version of BERT (MMBT) with experiments on text-only, image-only, and combined modalities, leverages PyTorch and Huggingface’s Transformer models, and includes visualization approaches (integrated gradients and attention via BertViz).
File Structure
• The top-level folders include:
– “MMBT/”: Contains the core model implementation (mmbt.py, image.py, mmbt_config.py, mmbt_utils.py). This is the central module that fuses text and image inputs.
– “data/”: Stores input data in various formats (CSV, JSON, image label files) plus pre-trained models. This is where the raw data – including image labels and metadata – is kept.
– “preprocess/”: Contains scripts to extract and filter labels from raw datasets (although the README notes this folder is not required to run the experiments, it still outlines the preprocessing step).
– “runs/”: Directory where Tensorboard and saved model outputs from experiments are stored.
– Notebooks (files with .ipynb extension, e.g., run_mmbt.ipynb, run_bert_text_only.ipynb): These cover the pipelines for experiments, training runs, and evaluation runs.
– Various requirement files and environment YAML files that specify dependencies and platform-specific settings.
README for Additional Insights
• The README details the overall experimental design and architectural decisions.
• It provides information on how the models, training parameters, and experiments (text-only, multimodal, masked text evaluation, and image-only from multimodal fine-tuning) are organized.
• It points out the dual-modality input (text tokens from radiology reports and image embeddings from X-rays) and their fusion within the MMBT model.
• It also explains that the implemented attention mechanisms (with comparisons via BertViz) and integrated gradients for explainability are key building blocks.
a. Main Components:
• Data Ingestion and Preprocessing:
• Data Sources: CSV, JSON, and image files in the “data/” directory.
• Preprocessing Scripts: Found in the “preprocess/” folder; these filter and label data.
• Model Core (MMBT):
• Text Encoder: BERT-based model (using Huggingface’s “BERT-base-uncased”).
• Image Submodel: (e.g., DenseNet-based, as referenced by the image_only experiments).
• Fusion Mechanism: The multimodal integration layer that aligns and fuses image embeddings with text token embeddings.
• Utility and Configuration: Modules like mmbt_config.py and mmbt_utils.py handle settings and helper functions.
• Experimentation & Training:
• Jupyter Notebooks: “run_mmbt.ipynb”, “run_bert_text_only.ipynb”, “run_mmbt_masked_text_eval.ipynb”, etc.
• Training Logs and Checkpoints: Stored in “runs/”.
• Hyperparameter Settings: Configurations and arguments in the notebooks and associated requirements.
• Visualization and Explainability:
• Integrated Gradients: Located in “integrated_gradients/”, to analyze model explanations.
• Attention Visualization: Implemented via BertViz (demonstrated in notebooks like “bertviz_attention.ipynb”).
b. Relationships and Interactions:
• Data flows from the “data/” directory to the preprocessing scripts, then through the model.
• The training pipeline (set up in the notebooks) loads data, calls the MMBT model (and variants) and logs results to “runs/”.
• The outputs of the MMBT model (predictions and attention scores) are visualized by downstream scripts (both integrated gradients and BertViz).
• External libraries (PyTorch, Huggingface, BertViz) interact with respective components (model training, data processing, and visualization).
c. Architectural Patterns and Design Principles:
• Modular Design: Separation into data ingestion, pre-processing, model core, experiment orchestration, and visualization modules.
• Pipeline Architecture: Clear path from raw data → preprocessing → model training → evaluation/visualization.
• Experimentation & Reproducibility: Several notebooks ensure reproducibility of experiments; results are logged in the “runs/” folder.
• Extensibility: The project is designed to allow swapping between text-only and multimodal components, with flexible configuration files.
d. Relevant Technologies and Frameworks:
• Programming Language: Python.
• Deep Learning Framework: PyTorch.
• Transformer Models: Huggingface’s BERT.
• Visualization Tools: BertViz for attention and custom integrated gradients visualization.
• Data Formats: CSV, JSON, and image files (with additional requirements to create directories for image data).
Instructing the Principal Software Engineer
• Provide comments or annotations on key design choices like the multimodal fusion approach and integration of external libraries.
• You capture important technical details such as framework usage (PyTorch, Huggingface Transformers) and experiment orchestration (via Jupyter notebooks).
• Remember to incorporate elements such as:
• Clear labels for each module/component