# porterlin/cxr-clip-cgmh architecture

> Architecture diagram of the GitHub repository porterlin/cxr-clip-cgmh, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/porterlin/cxr-clip-cgmh
- Repository: https://github.com/porterlin/cxr-clip-cgmh

## Overview

##### Project type and purpose
- This is a machine‐learning system for large‐scale chest X-ray language-image pre‐training (“CXR-CLIP-CGMH”), implemented in Python using PyTorch and Hydra.
- It supports:
- Zero-shot image–text contrastive pre-training (CLIP-style)
- Linear probing fine-tuning on downstream tasks (classification)
- Evaluation (zero-shot, fine-tune) and single-image inference

##### File structure
- Top‐level scripts (entrypoints):
- train.py (pre-training)
- finetune.py (linear probe training)
- evaluate_clip.py (zero-shot eval)
- evaluate_finetune.py (fine-tune eval)
- inference.py (single image)
- Configuration (Hydra + YAML):
- configs/: data_train / data_valid / data_test / dataloader / model / optimizer / scheduler / loss / tokenizer / transform / train.yaml
- Core code module (cxrclip/):
- cxrclip/data: data module, datasets, transforms, tokenization, prompt generation
- cxrclip/model: image_encoder, text_encoder, projection heads, classification module
- cxrclip/loss: contrastive loss, classification loss, combined loss
- cxrclip/optimizer & scheduler: custom wrappers (warmup cosine)
- cxrclip/prompt: constants & prompt templates
- cxrclip/trainer: orchestrates forward/backward, logging, checkpointing, distributed
- cxrclip/util: distributed utils, summary writer, global env
- datasets/: static JSON/README for prompt definitions
- text_augmentation/: optional back-translation for text
- requirements.txt, .gitignore, LICENSE

##### README insights
- Uses Hydra to modularly select data splits, models, optimizers, schedulers, losses.
- Supports single-GPU and multi-GPU (torchrun + NCCL)
- Prompts are loaded from JSON, can sample positive/negative/uncertain templates

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI / Entrypoint Layer
    subgraph "CLI / Entrypoint"
        direction TB
        train["train.py"]:::cli
        finetune["finetune.py"]:::cli
        eval_clip_script["evaluate_clip.py"]:::cli
        eval_finetune_script["evaluate_finetune.py"]:::cli
        inference_script["inference.py"]:::cli
    end

    %% Configuration Layer
    subgraph "Configuration"
        direction TB
        config["Hydra Configs (configs/)"]:::config
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        datamodule["DataModule\n(cxrclip/data/datamodule.py)"]:::data
        ds_imgcls["ImageClassification Dataset\n(cxrclip/data/datasets/image_classification.py)"]:::data
        ds_it["ImageText Dataset\n(cxrclip/data/datasets/imagetext.py)"]:::data
        ds_it_eval["ImageTextEval Dataset\n(cxrclip/data/datasets/imagetext_eval.py)"]:::data
        transform_def["Transform: default.yaml"]:::data
        transform_clahe["Transform: clahe.yaml"]:::data
        prompt_consts["Constants\n(cxrclip/prompt/constants.py)"]:::data
        prompt_impl["Prompts\n(cxrclip/prompt/prompts.py)"]:::data
        prompts_json["train_prompts_*.json"]:::data
        texts_json["texts.json"]:::data
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        model_clip["CLIP Model\n(cxrclip/model/clip.py)"]:::model
        model_imgcls["ImageClassification Model\n(cxrclip/model/image_classification.py)"]:::model
        enc_img["ImageEncoder\n(cxrclip/model/modules/image_encoder.py)"]:::model
        enc_text["TextEncoder\n(cxrclip/model/modules/text_encoder.py)"]:::model
        proj["Projection Heads\n(cxrclip/model/modules/projection.py)"]:::model
        classifier["Classifier Head\n(cxrclip/model/modules/image_classifier.py)"]:::model
    end

    %% Loss Layer
    subgraph "Loss Layer"
        direction TB
        loss_contrast["ContrastiveLoss\n(cxrclip/loss/cxr_clip.py)"]:::loss
        loss_class["ClassificationLoss\n(cxrclip/loss/classification.py)"]:::loss
        loss_comb["CombinedLoss\n(cxrclip/loss/combined_loss.py)"]:::loss
    end

    %% Optimization Layer
    subgraph "Optimization Layer"
        direction TB
        opt_adamw["AdamW Config\n(configs/optimizer/adamw.yaml)"]:::opt
        opt_sgd["SGD Config\n(configs/optimizer/sgd.yaml)"]:::opt
        sched_py["WarmupCosine\n(cxrclip/scheduler/warmup_cosine.py)"]:::opt
        sched_cfg["Scheduler Configs\n(configs/scheduler/*.yaml)"]:::opt
    end

    %% Training Orchestration & Utilities
    subgraph "Training & Utilities"
        direction TB
        trainer["Trainer\n(cxrclip/trainer.py)"]:::train
        dist_auto["DistAutograd\n(cxrclip/util/dist_autograd.py)"]:::train
        dist_sw["DistSummaryWriter\n(cxrclip/util/dist_summery_writer.py)"]:::train
        global_env["GlobalEnv\n(cxrclip/util/global_env.py)"]:::train
        utils["Utils\n(cxrclip/util/utils.py)"]:::train
    end

    %% Evaluation / Inference Layer
    subgraph "Evaluation / Inference"
        direction TB
        eval_clip["evaluate_clip.py"]:::eval
        eval_finetune["evaluate_finetune.py"]:::eval
        inference["inference.py"]:::eval
        evaluator["Evaluator\n(cxrclip/evaluator.py)"]:::eval
    end

    %% Data Flow
    train -->|uses| config
    finetune -->|uses| config
    eval_clip_script -->|uses| config
    eval_finetune_script -->|uses| config
    inference_script -->|uses| config

    config -->|instantiates| datamodule
    datamodule -->|loads| ds_imgcls
    datamodule -->|loads| ds_it
    datamodule -->|loads| ds_it_eval
    datamodule -->|applies| transform_def
    datamodule -->|applies| transform_clahe
    datamodule -->|generates| prompt_consts
    datamodule -->|generates| prompt_impl
    prompt_impl -->|reads| prompts_json
    prompt_impl -->|reads| texts_json

    ds_imgcls -->|batches| model_clip
    ds_it -->|batches| model_clip
    ds_it_eval -->|batches| model_clip

    model_clip --> enc_img
    model_clip --> enc_text
    enc_img --> proj
    enc_text --> proj
    proj --> loss_contrast
    model_imgcls --> classifier
    classifier --> loss_class
    loss_contrast --> loss_comb
    loss_class --> loss_comb

    loss_comb --> trainer
    opt_adamw --> trainer
    opt_sgd --> trainer
    sched_py --> trainer
    sched_cfg --> trainer

    trainer --> dist_auto
    trainer --> dist_sw
    trainer --> global_env
    trainer --> utils

    %% Evaluation & Inference Flows
    eval_clip --> datamodule
    eval_clip --> model_clip
    eval_clip --> evaluator
    eval_finetune --> datamodule
    eval_finetune --> model_imgcls
    eval_finetune --> evaluator
    inference --> datamodule
    inference --> model_clip

    %% Click Events
    click train "https://github.com/porterlin/cxr-clip-cgmh/blob/main/train.py"
    click finetune "https://github.com/porterlin/cxr-clip-cgmh/blob/main/finetune.py"
    click eval_clip_script "https://github.com/porterlin/cxr-clip-cgmh/blob/main/evaluate_clip.py"
    click eval_finetune_script "https://github.com/porterlin/cxr-clip-cgmh/blob/main/evaluate_finetune.py"
    click inference_script "https://github.com/porterlin/cxr-clip-cgmh/blob/main/inference.py"
    click config "https://github.com/porterlin/cxr-clip-cgmh/tree/main/configs/"
    click datamodule "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/data/datamodule.py"
    click ds_imgcls "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/data/datasets/image_classification.py"
    click ds_it "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/data/datasets/imagetext.py"
    click ds_it_eval "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/data/datasets/imagetext_eval.py"
    click transform_def "https://github.com/porterlin/cxr-clip-cgmh/blob/main/configs/transform/default.yaml"
    click transform_clahe "https://github.com/porterlin/cxr-clip-cgmh/blob/main/configs/transform/clahe.yaml"
    click prompt_consts "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/prompt/constants.py"
    click prompt_impl "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/prompt/prompts.py"
    click prompts_json "https://github.com/porterlin/cxr-clip-cgmh/blob/main/datasets/train_prompts_all.json"
    click texts_json "https://github.com/porterlin/cxr-clip-cgmh/blob/main/datasets/texts.json"
    click model_clip "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/model/clip.py"
    click model_imgcls "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/model/image_classification.py"
    click enc_img "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/model/modules/image_encoder.py"
    click enc_text "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/model/modules/text_encoder.py"
    click proj "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/model/modules/projection.py"
    click classifier "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/model/modules/image_classifier.py"
    click loss_contrast "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/loss/cxr_clip.py"
    click loss_class "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/loss/classification.py"
    click loss_comb "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/loss/combined_loss.py"
    click opt_adamw "https://github.com/porterlin/cxr-clip-cgmh/blob/main/configs/optimizer/adamw.yaml"
    click opt_sgd "https://github.com/porterlin/cxr-clip-cgmh/blob/main/configs/optimizer/sgd.yaml"
    click sched_py "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/scheduler/warmup_cosine.py"
    click sched_cfg "https://github.com/porterlin/cxr-clip-cgmh/tree/main/configs/scheduler/"
    click trainer "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/trainer.py"
    click eval_clip "https://github.com/porterlin/cxr-clip-cgmh/blob/main/evaluate_clip.py"
    click eval_finetune "https://github.com/porterlin/cxr-clip-cgmh/blob/main/evaluate_finetune.py"
    click inference "https://github.com/porterlin/cxr-clip-cgmh/blob/main/inference.py"
    click evaluator "https://github.com/porterlin/cxr-clip-cgmh/blob/main/cxrclip/evaluator.py"

    %% Styles
    classDef cli fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef config fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef data fill:#A8E6A2,stroke:#000,stroke-width:1px
    classDef model fill:#F4A261,stroke:#000,stroke-width:1px
    classDef loss fill:#E76F51,stroke:#000,stroke-width:1px
    classDef opt fill:#E76F51,stroke:#000,stroke-width:1px
    classDef train fill:#9D94FF,stroke:#000,stroke-width:1px
    classDef eval fill:#CCCCCC,stroke:#000,stroke-width:1px
```
