# sarahesl/pubmedclip architecture

> Architecture diagram of the GitHub repository sarahesl/pubmedclip, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sarahesl/pubmedclip
- Repository: https://github.com/sarahesl/pubmedclip

## Overview

##### Project type and purpose
- This is a research‐oriented deep learning pipeline for Medical Visual Question Answering (VQA), centered on a fine‐tuned CLIP model (“PubMedCLIP”).
- It provides three related codebases:
- PubMedCLIP: the core pre‐trained visual–text encoder with configs, model code, data loaders, training and evaluation scripts.
- MEVF_PubMedCLIP: integration of PubMedCLIP into the MEVF VQA pipeline.
- QCR_PubMedCLIP: integration into the QCR VQA pipeline, with additional networks and utilities.

##### File structure
- Top‐level directories:
- MEVF_PubMedCLIP/ – Python modules (attention, auto_encoder, classifier, trainer, etc.), run scripts, train/test entry points.
- PubMedCLIP/ –
– configs/ (YAML experiment settings)
– lib/ (subpackages: config, core, data_transform, dataset, utils)
– main/ (CLI entry points: create_jsons, train, main)
– requirements.txt, run.sh
- QCR_PubMedCLIP/ –
– configs/ (many YAMLs for different setups)
– lib/ (BAN models, config, core, dataset, language, network, utils)
– main/ (train/test scripts), run_*.sh, saved_models/
- Patterns: modular ML pipeline, separation of config, data, core logic, and entry‐point scripts.

##### README for additional insights
- Project goal: fine-tune CLIP on medical image-caption (ROCO) and evaluate gains in VQA tasks.
- Pipelines provided for MEVF and QCR.

##### Detailed capture of essential elements
- YAML config files feeding into both PubMedCLIP and pipeline subprojects
- Separate lanes for each subproject (PubMedCLIP, MEVF, QCR) to show reuse of core encoder

## Diagram (Mermaid)

```mermaid
graph TD
    %% Configurations
    subgraph "Configurations"
        ConfigPub["PubMedCLIP configs"]:::config
        ConfigQCR["QCR_PubMedCLIP configs"]:::config
    end

    %% Data Pipeline
    subgraph "Data Pipeline"
        ROCO["ROCOdataset"]:::data
        RAD["dataset_RAD"]:::data
        SLAKE["dataset_SLAKE"]:::data
        Transform["Transform Wrapper"]:::data
    end

    %% Core Model
    subgraph "Core Model"
        Encoder["PubMedCLIP Encoder"]:::model
        DefaultConfig["Encoder Hyperparams"]:::model
    end

    %% MEVF Task Heads
    subgraph "MEVF_PubMedCLIP Heads"
        MEVF_Att["attention"]:::model
        MEVF_AE["auto_encoder"]:::model
        MEVF_BC["bc"]:::model
        MEVF_Cls["classifier"]:::model
        MEVF_Count["counting"]:::model
        MEVF_FC["fc"]:::model
        MEVF_Lang["language_model"]:::model
    end

    %% QCR Task Heads
    subgraph "QCR_PubMedCLIP Heads"
        BAN["multi_level_model"]:::model
        QAE["auto_encoder"]:::model
        QCount["counting"]:::model
        QNet["network"]:::model
        QLang["language_model"]:::model
        QQ["classify_question"]:::model
    end

    %% Training & Evaluation
    subgraph "Training Engine & Utils"
        Trainer["Trainer & Loss"]:::orchestration
        Meters["Meters & Utils"]:::orchestration
        Registry["Registry"]:::orchestration
        CoreEval["Evaluation Module"]:::orchestration
    end

    %% Entry Points
    subgraph "CLI Entry Points"
        CreateJSON["create_jsons"]:::orchestration
        TrainPub["train"]:::orchestration
        MainPub["main"]:::orchestration
        MEVF_Test["test (MEVF)"]:::orchestration
        QCR_Train["train (QCR)"]:::orchestration
        QCR_Test["test (QCR)"]:::orchestration
    end

    %% Orchestration Scripts
    subgraph "Orchestration Scripts"
        RunMEVF1["run_clipOrg.sh"]:::orchestration
        RunMEVF2["run_pubmedclip.sh"]:::orchestration
        RunPub["run.sh"]:::orchestration
        RunQCR1["run_clipOrg_ae.sh"]:::orchestration
        RunQCR2["run_pubmedclip_ae.sh"]:::orchestration
    end

    %% Outputs
    subgraph "Outputs"
        Saved["saved_models"]:::output
        JSONs["Generated JSONs"]:::output
    end

    %% Relationships
    ConfigPub -->|feeds| ROCO
    ConfigPub -->|feeds| Transform
    ConfigQCR -->|feeds| RAD
    ConfigQCR -->|feeds| SLAKE

    ROCO --> Transform
    RAD --> Transform
    SLAKE --> Transform

    Transform --> Encoder
    ConfigPub --> DefaultConfig
    DefaultConfig --> Encoder

    Encoder --> MEVF_Att
    Encoder --> MEVF_AE
    Encoder --> MEVF_BC
    Encoder --> MEVF_Cls
    Encoder --> MEVF_Count
    Encoder --> MEVF_FC
    Encoder --> MEVF_Lang

    Encoder --> BAN
    Encoder --> QAE
    Encoder --> QCount
    Encoder --> QNet
    Encoder --> QLang
    Encoder --> QQ

    MEVF_Att --> Trainer
    MEVF_AE --> Trainer
    MEVF_BC --> Trainer
    MEVF_Cls --> Trainer
    MEVF_Count --> Trainer
    MEVF_FC --> Trainer
    MEVF_Lang --> Trainer

    BAN --> Trainer
    QAE --> Trainer
    QCount --> Trainer
    QNet --> Trainer
    QLang --> Trainer
    QQ --> Trainer

    Trainer --> Meters
    Trainer --> Registry
    Trainer --> CoreEval

    CoreEval --> CreateJSON
    CoreEval --> MEVF_Test
    CoreEval --> QCR_Test

    CreateJSON --> JSONs
    TrainPub --> Trainer
    QCR_Train --> Trainer
    MainPub --> Trainer

    RunMEVF1 --> TrainPub
    RunMEVF2 --> TrainPub
    RunPub --> MainPub
    RunQCR1 --> QCR_Train
    RunQCR2 --> QCR_Train

    Trainer --> Saved

    %% Click Events
    click ConfigPub "https://github.com/sarahesl/pubmedclip/blob/main/PubMedCLIP/configs/*.yaml"
    click ConfigQCR "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/configs/*.yaml"
    click ROCO "https://github.com/sarahesl/pubmedclip/blob/main/PubMedCLIP/lib/dataset/ROCOdataset.py"
    click RAD "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/lib/dataset/dataset_RAD.py"
    click SLAKE "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/lib/dataset/dataset_SLAKE.py"
    click Transform "https://github.com/sarahesl/pubmedclip/blob/main/PubMedCLIP/lib/data_transform/transform_wrapper.py"
    click Encoder "https://github.com/sarahesl/pubmedclip/blob/main/PubMedCLIP/lib/core/function.py"
    click DefaultConfig "https://github.com/sarahesl/pubmedclip/blob/main/PubMedCLIP/lib/config/default.py"
    click MEVF_Att "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/attention.py"
    click MEVF_AE "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/auto_encoder.py"
    click MEVF_BC "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/bc.py"
    click MEVF_Cls "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/classifier.py"
    click MEVF_Count "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/counting.py"
    click MEVF_FC "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/fc.py"
    click MEVF_Lang "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/language_model.py"
    click BAN "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/lib/BAN/multi_level_model.py"
    click QAE "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/lib/network/auto_encoder.py"
    click QCount "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/lib/network/counting.py"
    click QNet "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/lib/network/network.py"
    click QLang "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/lib/language/language_model.py"
    click QQ "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/lib/language/classify_question.py"
    click Trainer "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/trainer.py"
    click Meters "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/meters.py"
    click Registry "https://github.com/sarahesl/pubmedclip/blob/main/PubMedCLIP/lib/utils/registry.py"
    click CoreEval "https://github.com/sarahesl/pubmedclip/blob/main/PubMedCLIP/lib/core/evaluate.py"
    click CreateJSON "https://github.com/sarahesl/pubmedclip/blob/main/PubMedCLIP/main/create_jsons.py"
    click TrainPub "https://github.com/sarahesl/pubmedclip/blob/main/PubMedCLIP/main/train.py"
    click MainPub "https://github.com/sarahesl/pubmedclip/blob/main/PubMedCLIP/main/main.py"
    click MEVF_Test "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/test.py"
    click QCR_Train "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/main/train.py"
    click QCR_Test "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/main/test.py"
    click RunMEVF1 "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/run_clipOrg.sh"
    click RunMEVF2 "https://github.com/sarahesl/pubmedclip/blob/main/MEVF_PubMedCLIP/run_pubmedclip.sh"
    click RunPub "https://github.com/sarahesl/pubmedclip/blob/main/PubMedCLIP/run.sh"
    click RunQCR1 "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/run_clipOrg_ae.sh"
    click RunQCR2 "https://github.com/sarahesl/pubmedclip/blob/main/QCR_PubMedCLIP/run_pubmedclip_ae.sh"
    click Saved "https://github.com/sarahesl/pubmedclip/tree/main/QCR_PubMedCLIP/saved_models/"

    %% Styles
    classDef config fill:#dddddd,stroke:#333,stroke-width:1px
    classDef data fill:#a2d5a2,stroke:#333,stroke-width:1px
    classDef model fill:#a2c4f4,stroke:#333,stroke-width:1px
    classDef orchestration fill:#f4b183,stroke:#333,stroke-width:1px
    classDef output fill:#d9ead3,stroke:#333,stroke-width:1px
```
