# omar-mohamed/gpt2-chest-x-ray-report-generation architecture

> Architecture diagram of the GitHub repository omar-mohamed/gpt2-chest-x-ray-report-generation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/omar-mohamed/gpt2-chest-x-ray-report-generation
- Repository: https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation

## Overview

##### Project type and purpose
- This is a machine-learning pipeline for automated radiology report generation.
- It conditions a pre-trained GPT-2 transformer on visual features (from a fine-tuned CheXNet CNN) and semantic features (from medical word embeddings) to produce chest X-ray reports.

##### File structure
- Top-level scripts:
- data acquisition & preprocessing: get_iu_xray.py, create_csv_from_xml.py, augmenter.py, count_anomaly_cases.py
- model training & inference: train.py, generator.py
- evaluation: caption_evaluation.py, get_scores_from_predictions.py, test.py
- utilities & configuration: configs.py, utility.py, tags.py
- Subdirectories:
- gpt2/ (gpt2_model.py, pretrained_model.py) – defines the conditioned transformer
- medical_word_embeddings/ – precomputed embeddings and loader wrapper
- pretrained_visual_model/ – fine_tuned_chexnet.* (model architecture + weights)
- Requirements files and README provide environment setup and usage.

##### Key components
a. Data Layer
- IU-XRay dataset (XMLs, CSVs)
- Scripts: get_iu_xray.py → create_csv_from_xml.py → training_set.csv & testing_set.csv
- Optional data augmentation: augmenter.py
b. Feature Extraction
- Visual Encoder
- pretrained_visual_model/fine_tuned_chexnet
- CNN_encoder.py wraps CheXNet for image → feature vector
- Semantic Encoder
- medical_word_embeddings/saved_embeddings.pickle
- medical_w2v_wrapper.py loads embeddings for disease tags
- tags.py extracts disease labels from metadata
c. Conditioned Transformer (Core Model)
- gpt2/pretrained_model.py loads base GPT-2
- gpt2/gpt2_model.py extends GPT-2 with cross-modal conditioning layers
- tokenizer_wrapper.py handles text tokenization
- configs.py centralizes hyperparameters (learning rate, batch size, model sizes)
d. Training & Inference Pipelines
- train.py ties together dataset loader, encoders, transformer model, loss functions
- generator.py runs inference to generate reports from new X-ray images
e. Evaluation & Utilities
- caption_evaluation.py computes text quality metrics
- get_scores_from_predictions.py aggregates evaluation scores
- utility.py houses shared helper functions
- test.py contains sanity checks / unit tests

##### Relationships and data flow
Dataset → Preprocessing → Feature Extraction (Visual + Semantic) → Conditioned GPT-2 → Generated Report
- Training loop: Model + loss → parameter updates → checkpointing
- Inference loop: Frozen model → generator.py → output text
- Evaluation: Generated reports + ground truth → evaluation scripts → metrics

##### Architectural patterns and design principles
- Modular separation of concerns: data prep, feature extraction, model definition, training, inference, evaluation
- Re-use of pretrained components (transfer learning for both CNN and transformer)
- Configuration-driven design (configs.py) for easy hyperparameter tuning
- Wrapper patterns around external libraries (tokenizer_wrapper.py, medical_w2v_wrapper.py)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        D1(("IU-XRay/")):::datastore
        A1["get_iu_xray.py"]:::script
        A2["create_csv_from_xml.py"]:::script
        CSV1(("training_set.csv")):::datastore
        CSV2(("testing_set.csv")):::datastore
        A3["augmenter.py"]:::script
        A4["count_anomaly_cases.py"]:::script
    end

    %% Feature Extraction
    subgraph "Feature Extraction"
        direction TB
        subgraph "Visual Encoder" 
            direction TB
            P1["pretrained_visual_model/fine_tuned_chexnet.json"]:::external
            P2["pretrained_visual_model/fine_tuned_chexnet.h5"]:::external
            V1["CNN_encoder.py"]:::process
        end
        subgraph "Semantic Encoder"
            direction TB
            S1["medical_word_embeddings/saved_embeddings.pickle"]:::datastore
            S2["medical_w2v_wrapper.py"]:::process
            S3["tags.py"]:::script
        end
    end

    %% Conditioned Transformer
    subgraph "Conditioned Transformer"
        direction TB
        T1["gpt2/pretrained_model.py"]:::external
        T2["gpt2/gpt2_model.py"]:::process
        T3["tokenizer_wrapper.py"]:::process
        T4["configs.py"]:::script
    end

    %% Pipelines
    subgraph "Pipelines"
        direction TB
        TR["train.py"]:::script
        GN["generator.py"]:::script
    end

    %% Evaluation & Utilities
    subgraph "Evaluation & Utilities"
        direction TB
        E1["caption_evaluation.py"]:::script
        E2["get_scores_from_predictions.py"]:::script
        U1["utility.py"]:::script
        U2["test.py"]:::script
    end

    %% Data Flow
    D1 -->|load XML| A1
    A1 -->|produce CSV| A2
    A2 --> CSV1
    A2 --> CSV2
    CSV1 -->|augment| A3
    CSV2 -->|analyze| A4

    %% Feature Extraction Flow
    CSV1 -->|image data| V1
    V1 -->|features| T2
    CSV1 -->|text tags| S3
    S3 --> S2
    S2 -->|embeddings| T2

    %% Transformer to Pipelines
    T1 --> T2
    T3 --> T2
    T4 --> T2

    T2 -->|model| TR
    V1 -->|visual features| TR
    S2 -->|semantic features| TR

    TR -->|trained model| T2
    TR -->|checkpoint| T1

    %% Inference
    V1 -->|visual features| GN
    S2 -->|semantic features| GN
    T2 -->|frozen model| GN

    %% Evaluation Flow
    GN -->|reports| E1
    E1 -->|metrics| E2
    E2 -->|results| U1
    U1 --> U2

    %% Click Events
    click D1 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/tree/master/IU-XRay/"
    click A1 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/get_iu_xray.py"
    click A2 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/create_csv_from_xml.py"
    click A3 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/augmenter.py"
    click A4 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/count_anomaly_cases.py"
    click V1 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/pretrained_visual_model/CNN_encoder.py"
    click P1 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/pretrained_visual_model/fine_tuned_chexnet.json"
    click P2 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/pretrained_visual_model/fine_tuned_chexnet.h5"
    click S2 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/medical_w2v_wrapper.py"
    click S1 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/medical_word_embeddings/saved_embeddings.pickle"
    click S3 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/tags.py"
    click T1 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/gpt2/pretrained_model.py"
    click T2 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/gpt2/gpt2_model.py"
    click T3 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/tokenizer_wrapper.py"
    click T4 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/configs.py"
    click TR "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/train.py"
    click GN "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/generator.py"
    click E1 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/caption_evaluation.py"
    click E2 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/get_scores_from_predictions.py"
    click U1 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/utility.py"
    click U2 "https://github.com/omar-mohamed/gpt2-chest-x-ray-report-generation/blob/master/test.py"

    %% Styles
    classDef datastore fill:#f9f,stroke:#333,stroke-width:1px,shape:cylinder
    classDef script fill:#bbf,stroke:#333,stroke-width:1px,shape:rect
    classDef process fill:#bfb,stroke:#333,stroke-width:1px,shape:round
    classDef external fill:#fdd,stroke:#933,stroke-dasharray: 5 5,stroke-width:1px
```
