How omar-mohamed/gpt2-chest-x-ray-report-generation works
Project type and purpose
• This is a machine-learning pipeline for automated radiology report generation.
• It conditions a pre-trained GPT-2 transformer on visual features (from a fine-tuned CheXNet CNN) and semantic features (from medical word embeddings) to produce chest X-ray reports.
File structure
• Top-level scripts:
• data acquisition & preprocessing: get_iu_xray.py, create_csv_from_xml.py, augmenter.py, count_anomaly_cases.py
• model training & inference: train.py, generator.py
• evaluation: caption_evaluation.py, get_scores_from_predictions.py, test.py
• utilities & configuration: configs.py, utility.py, tags.py
• Subdirectories:
• gpt2/ (gpt2_model.py, pretrained_model.py) – defines the conditioned transformer
• medical_word_embeddings/ – precomputed embeddings and loader wrapper
• pretrained_visual_model/ – fine_tuned_chexnet.* (model architecture + weights)
• Requirements files and README provide environment setup and usage.
Key components
a. Data Layer
• IU-XRay dataset (XMLs, CSVs)
• Scripts: get_iu_xray.py → create_csv_from_xml.py → training_set.csv & testing_set.csv
• Optional data augmentation: augmenter.py
b. Feature Extraction
• Visual Encoder
• pretrained_visual_model/fine_tuned_chexnet
• CNN_encoder.py wraps CheXNet for image → feature vector
• Semantic Encoder
• medical_word_embeddings/saved_embeddings.pickle
• medical_w2v_wrapper.py loads embeddings for disease tags
• tags.py extracts disease labels from metadata
c. Conditioned Transformer (Core Model)
• gpt2/pretrained_model.py loads base GPT-2
• gpt2/gpt2_model.py extends GPT-2 with cross-modal conditioning layers
• tokenizer_wrapper.py handles text tokenization
• configs.py centralizes hyperparameters (learning rate, batch size, model sizes)
d. Training & Inference Pipelines
• train.py ties together dataset loader, encoders, transformer model, loss functions
• generator.py runs inference to generate reports from new X-ray images
e. Evaluation & Utilities
• caption_evaluation.py computes text quality metrics
• get_scores_from_predictions.py aggregates evaluation scores
• utility.py houses shared helper functions
• test.py contains sanity checks / unit tests
Relationships and data flow
Dataset → Preprocessing → Feature Extraction (Visual + Semantic) → Conditioned GPT-2 → Generated Report
• Training loop: Model + loss → parameter updates → checkpointing
• Inference loop: Frozen model → generator.py → output text
• Evaluation: Generated reports + ground truth → evaluation scripts → metrics
Architectural patterns and design principles
• Modular separation of concerns: data prep, feature extraction, model definition, training, inference, evaluation
• Re-use of pretrained components (transfer learning for both CNN and transformer)
• Configuration-driven design (configs.py) for easy hyperparameter tuning
• Wrapper patterns around external libraries (tokenizer_wrapper.py, medical_w2v_wrapper.py)