How batmanlab/mammo-clip works
Project Type and Purpose
• The project is a vision–language framework for mammography. It is a research-focused deep learning system that pretrains a foundation model (Mammo‑CLIP) on image–text and image–label data for tasks including classification, detection, and zero‑shot evaluation.
• The README outlines preprocessing of mammography images (from RSNA and VinDr datasets), text augmentation from radiology reports, pretraining configurations, downstream tasks, and evaluation pipelines. This indicates that the system is a full-stack deep learning project with well-separated stages (data ingestion, preprocessing, training/finetuning, and evaluation).
Analyzing the File Structure
• The top-level directories (e.g., “src”, “preprocessing”, “scripts”, “notebooks”) indicate separation of concerns.
– “Classifiers”: Implements classes (e.g., BreastClipClassifier) for downstream classification tasks.
– “Detectors”: Contains components for detection models (e.g., RetinaNet implementations under “retinanet”).
– “Datasets” and “breastclip/data”: Handle data ingestion and organization.
– “configs”: Holds YAML configuration files for training, optimizer, scheduler, and dataloader setups.
– “notebooks” offer interactive tutorials and examples.
– “scripts” include command-line execution scripts for training and evaluation.
• This modular structure reflects an architectural choice that separates core algorithms (model, loss, optimizer) from data management and training orchestration.
Key Components
a. Main components:
• Data Preprocessing & Ingestion: Scripts under “src/preprocessing” and dataset-related modules.
• Image–Text and Image–Label Data Preparation: CSV processing (e.g., augment_text.py) and structured data (upmc and VinDr datasets).
• Core Model Modules: The backbone modules in “breastclip/model” and “Detectors/retinanet” (Efficient-Net implementations, CLIP components).
• Training Orchestration: Trainer modules (e.g., train.py, train_classifier.py, train_detector.py) and configuration management using YAML.
• Evaluation and Inference: Evaluation scripts (e.g., eval_zero_shot_clip.py) and notebooks for downstream tasks.
• Auxiliary Components: Utility functions, metrics, and logging modules in “src/codebase/utils.py” and “breastclip/util”.
b. Relationships & interactions:
• Data flows from raw images and CSVs (preprocessing layer) into dataset modules which then feed into Training pipelines.
• The model training component uses configurations (loaded from YAML), utilities, optimizers, and loss functions—all orchestrated by trainer scripts.
• Downstream tasks use the pretrained backbone in classifier and detector modules.
• Evaluation components receive checkpoints and run tests using a defined interface (e.g., through the command‐line evaluation scripts).
c. Architectural patterns/design principles:
• Modular separation: Each component (data, model, training, evaluation) is clearly split into its own (sub)module.
• Config-driven design: YAML files in “configs” allow for easy experimentation and hyperparameter tuning.
• Reusability and extensibility: With separate directories for classifiers, detectors, and preprocessing, new tasks or data sources can be added with minimal friction.
d. Relevant tooling & technologies:
• Python (3.8.18) and PyTorch (2.2.2) as core framework.
• CUDA (11.8) for GPU acceleration.
• Conda environment management (environment.yml) to ensure dependency control.
• Supporting tools: Hydra for config management (see command-line overrides in training/evaluation scripts).
Step-by-Step Guide: How to Use It Locally
a. Clone and Set Up Environment:
1. Clone the repository:
git clone git@github.com:batmanlab/Mammo-CLIP.git
2. Navigate into the repository folder:
cd Mammo-CLIP
3. Create the Conda environment using the provided environment.yml:
conda env create --name Mammo-CLIP -f environment.yml
4. Activate the Conda environment:
conda activate Mammo-CLIP
b. Preprocess Your Data:
• Depending on your local data (RSNA or VinDr), run the appropriate preprocessing scripts.
• For example, to convert RSNA DICOM images to PNG:
python ./src/preprocessing/preprocess_image_to_png_kaggle.py --phase="test" --base_folder="your_local_path_to_RSNA_dataset"
• Similarly, run the VinDr preprocessing script with your own base folder path.
c. Prepare Datasets and Augment Text:
• If using the image–text dataset, run the augmentation command:
python ./src/codebase/augment_text.py --dataset-path="your_local_path/src/codebase/data_csv" --csv-path="upmc_dicom_consolidated_final_folds_BIRADS_num_1_report.csv" --dataset="upmc"
• Verify that the CSV outputs (clip_pretrain_100.csv, and clip_vindr_final.csv) have the correct paths.
d. Run Pretraining or Downstream Tasks:
• For pretraining the Mammo‑CLIP model (e.g., using Efficient-Net B5 on a single GPU):
python ./src/codebase/train.py --config-name pre_train_b5_clip.yaml
• For downstream evaluations, use provided classifier or detector training scripts. Adjust the parameters (paths, dataset names) to match your local setup.
e. Evaluate and Experiment:
• Use the evaluation scripts (e.g., eval_zero_shot_clip.py) to test model performance.