How zaaachos/thesis-diagnostic-captioning works
Overall Project Context & Components
• Three main operational layers:
a. Data Layer: Represents datasets (e.g., IU X-Ray in the “data” directory and other potential ImageCLEF datasets with different JSON and pkl files).
b. Model & Processing Layer: This includes various deep learning models and pipelines. Inside this layer, capture:
• The different model implementations found in “models” (for CNN-RNN, kNN, and ClinicalBERT).
• The “modules” which implement various search strategies (beam search, greedy search, cluster, etc.) – these serve as the decoding or inference components.
• The “SOTA_models” folder, which contains submodules for state-of-the-art approaches like R2Gen and VisualGPT. These are experimental and meant for comparative evaluation.
c. Utility & Evaluation Layer: Include the “utils” components which encapsulate common functions for data handling, metrics calculation, and caption evaluation (via pycocoevalcap and its BLEU, Cider, Meteor, Rouge blocks).
Data Flow and Relationships
• How raw input data from the “data” directory feeds into the pipeline.
• Preprocessing step (e.g., handled in utils/dataset.py and possible custom functions in utils/text_handler.py and utils/vocabulary.py).
• Finally, include the evaluation process where generated captions are compared with ground truth using the caption evaluation utilities in “utils/pycocoevalcap”.
Architectural Patterns & Key Technologies
• Architectural Style:
• The design is modular. Each core functionality (data reading, model definitions, and evaluation) is separated into its respective directory which suggests a layered approach.
• There’s an element of plug-and-play as reflected by the different experimental scripts (e.g., the various *.sh scripts for cross-modal and multi-modal configurations).
• Technologies:
• Python is the primary language.
• Additional libraries for evaluation such as the pycocoevalcap submodules for BLEU, CIDEr, Meteor, and ROUGE.
• Data Sources: “data” directory (with IU X-Ray dataset example & extra pre-trained resources like FastText files).
• Preprocessing: Data loading and text handling modules (found in “utils”).
• Inference/Decoding: Modules for beam and greedy search from “modules”.
• Evaluation: “utils/pycocoevalcap”.
• Scripts/Execution: “dc.py” as the main entry point and the specific shell scripts for different training procedures.
• Input from data to preprocessing.
• The flow from preprocessing to model training.
• How the output from models is evaluated.
• A notation where the environment setup decisions (TensorFlow-GPU vs. PyTorch environment) affect the flow into the “models” layer. This is especially important for differentiating between ClinicalBERT and other model flows.