# batmanlab/mammo-clip architecture

> Architecture diagram of the GitHub repository batmanlab/mammo-clip, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/batmanlab/mammo-clip
- Repository: https://github.com/batmanlab/mammo-clip

## Overview

##### Project Type and Purpose
- The project is a vision–language framework for mammography. It is a research-focused deep learning system that pretrains a foundation model (Mammo‑CLIP) on image–text and image–label data for tasks including classification, detection, and zero‑shot evaluation.
- The README outlines preprocessing of mammography images (from RSNA and VinDr datasets), text augmentation from radiology reports, pretraining configurations, downstream tasks, and evaluation pipelines. This indicates that the system is a full-stack deep learning project with well-separated stages (data ingestion, preprocessing, training/finetuning, and evaluation).

##### Analyzing the File Structure
- The top-level directories (e.g., “src”, “preprocessing”, “scripts”, “notebooks”) indicate separation of concerns.
– “Classifiers”: Implements classes (e.g., BreastClipClassifier) for downstream classification tasks.
– “Detectors”: Contains components for detection models (e.g., RetinaNet implementations under “retinanet”).
– “Datasets” and “breastclip/data”: Handle data ingestion and organization.
– “configs”: Holds YAML configuration files for training, optimizer, scheduler, and dataloader setups.
– “notebooks” offer interactive tutorials and examples.
– “scripts” include command-line execution scripts for training and evaluation.
- This modular structure reflects an architectural choice that separates core algorithms (model, loss, optimizer) from data management and training orchestration.

##### Key Components
a. Main components:
- Data Preprocessing & Ingestion: Scripts under “src/preprocessing” and dataset-related modules.
- Image–Text and Image–Label Data Preparation: CSV processing (e.g., augment_text.py) and structured data (upmc and VinDr datasets).
- Core Model Modules: The backbone modules in “breastclip/model” and “Detectors/retinanet” (Efficient-Net implementations, CLIP components).
- Training Orchestration: Trainer modules (e.g., train.py, train_classifier.py, train_detector.py) and configuration management using YAML.
- Evaluation and Inference: Evaluation scripts (e.g., eval_zero_shot_clip.py) and notebooks for downstream tasks.
- Auxiliary Components: Utility functions, metrics, and logging modules in “src/codebase/utils.py” and “breastclip/util”.

b. Relationships & interactions:
- Data flows from raw images and CSVs (preprocessing layer) into dataset modules which then feed into Training pipelines.
- The model training component uses configurations (loaded from YAML), utilities, optimizers, and loss functions—all orchestrated by trainer scripts.
- Downstream tasks use the pretrained backbone in classifier and detector modules.
- Evaluation components receive checkpoints and run tests using a defined interface (e.g., through the command‐line evaluation scripts).

c. Architectural patterns/design principles:
- Modular separation: Each component (data, model, training, evaluation) is clearly split into its own (sub)module.
- Config-driven design: YAML files in “configs” allow for easy experimentation and hyperparameter tuning.
- Reusability and extensibility: With separate directories for classifiers, detectors, and preprocessing, new tasks or data sources can be added with minimal friction.

d. Relevant tooling & technologies:
- Python (3.8.18) and PyTorch (2.2.2) as core framework.
- CUDA (11.8) for GPU acceleration.
- Conda environment management (environment.yml) to ensure dependency control.
- Supporting tools: Hydra for config management (see command-line overrides in training/evaluation scripts).

##### Step-by-Step Guide: How to Use It Locally
a. Clone and Set Up Environment:
1. Clone the repository:
git clone git@github.com:batmanlab/Mammo-CLIP.git
2. Navigate into the repository folder:
cd Mammo-CLIP
3. Create the Conda environment using the provided environment.yml:
conda env create --name Mammo-CLIP -f environment.yml
4. Activate the Conda environment:
conda activate Mammo-CLIP

b. Preprocess Your Data:
- Depending on your local data (RSNA or VinDr), run the appropriate preprocessing scripts.
- For example, to convert RSNA DICOM images to PNG:
python ./src/preprocessing/preprocess_image_to_png_kaggle.py --phase="test" --base_folder="your_local_path_to_RSNA_dataset"
- Similarly, run the VinDr preprocessing script with your own base folder path.

c. Prepare Datasets and Augment Text:
- If using the image–text dataset, run the augmentation command:
python ./src/codebase/augment_text.py --dataset-path="your_local_path/src/codebase/data_csv" --csv-path="upmc_dicom_consolidated_final_folds_BIRADS_num_1_report.csv" --dataset="upmc"
- Verify that the CSV outputs (clip_pretrain_100.csv, and clip_vindr_final.csv) have the correct paths.

d. Run Pretraining or Downstream Tasks:
- For pretraining the Mammo‑CLIP model (e.g., using Efficient-Net B5 on a single GPU):
python ./src/codebase/train.py --config-name pre_train_b5_clip.yaml
- For downstream evaluations, use provided classifier or detector training scripts. Adjust the parameters (paths, dataset names) to match your local setup.

e. Evaluate and Experiment:
- Use the evaluation scripts (e.g., eval_zero_shot_clip.py) to test model performance.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer: Preprocessing & Ingestion
    subgraph "Data Preprocessing & Ingestion"
        PreRSNA["Preprocess RSNA"]:::datalayer
        PreVinDrDet["Preprocess VinDr Detector"]:::datalayer
        PreVinDrPNG["Preprocess VinDr PNG"]:::datalayer
        Datasets["Datasets"]:::datalayer
        BreastclipData["Breastclip Data"]:::datalayer
    end

    %% Data Preparation: Image-Text & Image-Label
    subgraph "Image-Text & Image-Label Data Preparation"
        TextAugPy["Text Augmentation (Python)"]:::dataprep
        TextAugSh["Text Augmentation (Shell)"]:::dataprep
        CSVData["CSV Data"]:::dataprep
    end

    %% Core Model Modules
    subgraph "Core Model Modules"
        BreastclipModel["Breastclip Model Components"]:::coremodel
        DetectionModels["Detection Models (RetinaNet)"]:::coremodel
    end

    %% Training Orchestration
    subgraph "Training Orchestration"
        Train["Train Script"]:::training
        TrainClassifier["Train Classifier Script"]:::training
        TrainDetector["Train Detector Script"]:::training
        ConfigMgmt["Configuration Management"]:::training
        DistTraining["Distributed Training"]:::training
    end

    %% Evaluation and Inference
    subgraph "Evaluation and Inference"
        EvalScript["Evaluation Script"]:::evaluation
        Notebooks["Notebooks"]:::evaluation
    end

    %% Auxiliary Components
    subgraph "Auxiliary Components"
        UtilFuncs["Utility Functions"]:::auxiliary
        BreastclipUtil["Breastclip Utilities"]:::auxiliary
        MetricsMod["Metrics Module"]:::auxiliary
    end

    %% Local Setup Instructions
    subgraph "Local Setup"
        CloneSetup["Clone & Environment Setup"]
        RunPreproc["Run Preprocessing"]
        PrepareData["Prepare Datasets & Augment Text"]
        RunTraining["Run Training"]
        RunEval["Run Evaluation"]
    end

    %% Connections between Data Preprocessing and Data Preparation
    PreRSNA --> Datasets
    PreVinDrDet --> Datasets
    PreVinDrPNG --> BreastclipData

    Datasets --> TextAugPy
    Datasets --> CSVData
    BreastclipData --> CSVData

    %% Connections to Training Orchestration
    TextAugPy --> Train
    TextAugSh --> Train
    CSVData --> Train
    Datasets --> Train
    BreastclipData --> Train

    %% Core Model Modules are used in training
    BreastclipModel --> Train
    DetectionModels --> TrainDetector

    %% Training Orchestration interconnections
    ConfigMgmt --> Train
    DistTraining --> Train
    Train --> TrainClassifier
    Train --> TrainDetector

    %% Auxiliary support for training
    UtilFuncs --- Train
    BreastclipUtil --- Train
    MetricsMod --- Train

    %% Evaluation Phase
    TrainClassifier --> EvalScript
    TrainDetector --> EvalScript
    EvalScript --> Notebooks

    %% Local Setup Flow
    CloneSetup --> RunPreproc
    RunPreproc --> PrepareData
    PrepareData --> RunTraining
    RunTraining --> RunEval

    %% Click Events for Data Preprocessing & Ingestion
    click PreRSNA "https://github.com/batmanlab/mammo-clip/blob/main/src/preprocessing/preprocess_image_to_png_kaggle.py"
    click PreVinDrDet "https://github.com/batmanlab/mammo-clip/blob/main/src/preprocessing/preprocess_VinDr_detector.py"
    click PreVinDrPNG "https://github.com/batmanlab/mammo-clip/blob/main/src/preprocessing/preprocess_image_to_png_vindr.py"
    click Datasets "https://github.com/batmanlab/mammo-clip/tree/main/src/codebase/Datasets"
    click BreastclipData "https://github.com/batmanlab/mammo-clip/tree/main/src/codebase/breastclip/data"

    %% Click Events for Data Preparation
    click TextAugPy "https://github.com/batmanlab/mammo-clip/blob/main/src/codebase/augment_text.py"
    click TextAugSh "https://github.com/batmanlab/mammo-clip/blob/main/src/scripts/augment_text.sh"
    click CSVData "https://github.com/batmanlab/mammo-clip/tree/main/src/codebase/data_csv"

    %% Click Events for Core Model Modules
    click BreastclipModel "https://github.com/batmanlab/mammo-clip/tree/main/src/codebase/breastclip/model"
    click DetectionModels "https://github.com/batmanlab/mammo-clip/tree/main/src/codebase/Detectors/retinanet"

    %% Click Events for Training Orchestration
    click Train "https://github.com/batmanlab/mammo-clip/blob/main/src/codebase/train.py"
    click TrainClassifier "https://github.com/batmanlab/mammo-clip/blob/main/src/codebase/train_classifier.py"
    click TrainDetector "https://github.com/batmanlab/mammo-clip/blob/main/src/codebase/train_detector.py"
    click ConfigMgmt "https://github.com/batmanlab/mammo-clip/tree/main/src/codebase/configs"
    click DistTraining "https://github.com/batmanlab/mammo-clip/blob/main/src/codebase/breastclip/trainer_ddp.py"

    %% Click Events for Evaluation and Inference
    click EvalScript "https://github.com/batmanlab/mammo-clip/blob/main/src/codebase/eval_zero_shot_clip.py"
    click Notebooks "https://github.com/batmanlab/mammo-clip/blob/main/src/codebase/notebooks/Tutorial/Downstream_classifier_tutorial.ipynb"

    %% Click Events for Auxiliary Components
    click UtilFuncs "https://github.com/batmanlab/mammo-clip/blob/main/src/codebase/utils.py"
    click BreastclipUtil "https://github.com/batmanlab/mammo-clip/tree/main/src/codebase/breastclip/util"
    click MetricsMod "https://github.com/batmanlab/mammo-clip/blob/main/src/codebase/metrics.py"

    %% Styling
    classDef datalayer fill:#cce5ff,stroke:#0056b3,stroke-width:2px;
    classDef dataprep fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef coremodel fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef training fill:#fde2e2,stroke:#dc3545,stroke-width:2px;
    classDef evaluation fill:#ffeeba,stroke:#856404,stroke-width:2px;
    classDef auxiliary fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
    classDef localsetup fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
```
