How sarahesl/pubmedclip works
Project type and purpose
• This is a research‐oriented deep learning pipeline for Medical Visual Question Answering (VQA), centered on a fine‐tuned CLIP model (“PubMedCLIP”).
• It provides three related codebases:
• PubMedCLIP: the core pre‐trained visual–text encoder with configs, model code, data loaders, training and evaluation scripts.
• MEVF_PubMedCLIP: integration of PubMedCLIP into the MEVF VQA pipeline.
• QCR_PubMedCLIP: integration into the QCR VQA pipeline, with additional networks and utilities.
File structure
• Top‐level directories:
• MEVF_PubMedCLIP/ – Python modules (attention, auto_encoder, classifier, trainer, etc.), run scripts, train/test entry points.
• PubMedCLIP/ –
– configs/ (YAML experiment settings)
– lib/ (subpackages: config, core, data_transform, dataset, utils)
– main/ (CLI entry points: create_jsons, train, main)
– requirements.txt, run.sh
• QCR_PubMedCLIP/ –
– configs/ (many YAMLs for different setups)
– lib/ (BAN models, config, core, dataset, language, network, utils)
– main/ (train/test scripts), run_*.sh, saved_models/
• Patterns: modular ML pipeline, separation of config, data, core logic, and entry‐point scripts.
README for additional insights
• Project goal: fine-tune CLIP on medical image-caption (ROCO) and evaluate gains in VQA tasks.
• Pipelines provided for MEVF and QCR.
Detailed capture of essential elements
• YAML config files feeding into both PubMedCLIP and pipeline subprojects
• Separate lanes for each subproject (PubMedCLIP, MEVF, QCR) to show reuse of core encoder