How jorwnpay/petrg-3d works
PETRG-3D trains and runs a multimodal model that turns paired 3D PET/CT scans into structured Chinese findings. Training pairs processed volumes with report text and optimizes the model; inference produces predictions in a resumable CSV. Evaluation can clean those reports, score text similarity, or extract anatomical labels through an OpenAI-compatible LLM and calculate clinical efficacy scores. README documentation establishes the evaluation stages; the inference implementation was not sampled, so its internal call wiring is omitted.
Main entry points: train.py Training entrypoint · test.py Inference entrypoint · my_embedding_layer.py Multimodal embeddings · PETRG_3D.py PETRG3D
Components 26
Data preparation
- Training dataset
src/Dataset/petct_dataset_train.py - Inference dataset
src/Dataset/petct_dataset_test.py - Training arguments
src/args/train/default.py - Inference arguments
src/args/test/default.py
PETRG-3D model
- PETRG3D
src/Model/PETRG_3D.py - Multimodal embeddings
src/Model/my_embedding_layer.py - 3D ViT backbone
src/Model/vit_3d.py - Position encoding
src/Model/position_encoding.py - Model helpers
src/Model/helpers.py
Training and inference
- Training entrypoint
src/train.py - Inference entrypoint
src/test.py
Report evaluation
- Report cleaning
evaluation/clean_csv_report.py - Chinese NLG metrics
evaluation/CN_nlg_evaluation.py - Local BLEU
evaluation/local_metrics/bleu/nmt_bleu.py - PET/CT label extraction
evaluation/extract_pet_ct_labels.py - Clinical efficacy scores
evaluation/cal_petrg_ces.py
Other components
- Researcher
- PET/CT and reports
- Chat LLM
- Visual checkpoint
- Prediction CSV
- Clean report JSONs
- LLM API
- Predicted labels
- Ground-truth labels
- Metric CSVs
Connections 30
- Researcher → Training entrypoint: configures and launches
- Researcher → Inference entrypoint: configures and launches
- PET/CT and reports → Training dataset: loads and preprocesses
- PET/CT and reports → Inference dataset: loads for inference
- Training entrypoint → Training arguments: parses arguments
- Training entrypoint → Training dataset: constructs
- Training entrypoint → PETRG3D: constructs and trains
- Training entrypoint → Visual checkpoint: loads optional weights
- PETRG3D → Multimodal embeddings: builds and calls
- PETRG3D → Chat LLM: wraps with LoRA and calls
- Multimodal embeddings → 3D ViT backbone: encodes volumes
- Multimodal embeddings → Visual checkpoint: loads visual weights
- 3D ViT backbone → Position encoding: uses encodings
- Multimodal embeddings → Model helpers: uses components
- Researcher → Prediction CSV: provides inference inputs
- Inference entrypoint → Inference arguments: parses arguments
- Inference entrypoint → Inference dataset: constructs
- Inference entrypoint → PETRG3D: generates reports
- Inference entrypoint → Prediction CSV: writes predictions
- Prediction CSV → Report cleaning: reads
- Report cleaning → Clean report JSONs: writes cleaned reports
- Prediction CSV → Chinese NLG metrics: supplies references and predictions
- Chinese NLG metrics → Local BLEU: computes BLEU
- Clean report JSONs → PET/CT label extraction: supplies findings
- PET/CT label extraction → LLM API: requests structured labels
- PET/CT label extraction → Predicted labels: writes label JSONs
- Predicted labels → Clinical efficacy scores: supplies predictions
- Ground-truth labels → Clinical efficacy scores: supplies reference labels
- Clinical efficacy scores → Metric CSVs: writes scores
- Chinese NLG metrics → Metric CSVs: reports text metrics