# zaaachos/thesis-diagnostic-captioning architecture

> Architecture diagram of the GitHub repository zaaachos/thesis-diagnostic-captioning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zaaachos/thesis-diagnostic-captioning
- Repository: https://github.com/zaaachos/thesis-diagnostic-captioning

## Overview

##### Overall Project Context & Components
- Three main operational layers:
a. Data Layer: Represents datasets (e.g., IU X-Ray in the “data” directory and other potential ImageCLEF datasets with different JSON and pkl files).
b. Model & Processing Layer: This includes various deep learning models and pipelines. Inside this layer, capture:
- The different model implementations found in “models” (for CNN-RNN, kNN, and ClinicalBERT).
- The “modules” which implement various search strategies (beam search, greedy search, cluster, etc.) – these serve as the decoding or inference components.
- The “SOTA_models” folder, which contains submodules for state-of-the-art approaches like R2Gen and VisualGPT. These are experimental and meant for comparative evaluation.
c. Utility & Evaluation Layer: Include the “utils” components which encapsulate common functions for data handling, metrics calculation, and caption evaluation (via pycocoevalcap and its BLEU, Cider, Meteor, Rouge blocks).

##### Data Flow and Relationships
- How raw input data from the “data” directory feeds into the pipeline.
- Preprocessing step (e.g., handled in utils/dataset.py and possible custom functions in utils/text_handler.py and utils/vocabulary.py).
- Finally, include the evaluation process where generated captions are compared with ground truth using the caption evaluation utilities in “utils/pycocoevalcap”.

##### Architectural Patterns & Key Technologies
- Architectural Style:
- The design is modular. Each core functionality (data reading, model definitions, and evaluation) is separated into its respective directory which suggests a layered approach.
- There’s an element of plug-and-play as reflected by the different experimental scripts (e.g., the various *.sh scripts for cross-modal and multi-modal configurations).
- Technologies:
- Python is the primary language.
- Additional libraries for evaluation such as the pycocoevalcap submodules for BLEU, CIDEr, Meteor, and ROUGE.

- Data Sources: “data” directory (with IU X-Ray dataset example & extra pre-trained resources like FastText files).
- Preprocessing: Data loading and text handling modules (found in “utils”).
- Inference/Decoding: Modules for beam and greedy search from “modules”.
- Evaluation: “utils/pycocoevalcap”.
- Scripts/Execution: “dc.py” as the main entry point and the specific shell scripts for different training procedures.
- Input from data to preprocessing.
- The flow from preprocessing to model training.
- How the output from models is evaluated.
- A notation where the environment setup decisions (TensorFlow-GPU vs. PyTorch environment) affect the flow into the “models” layer. This is especially important for differentiating between ClinicalBERT and other model flows.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Layer
    subgraph "Data Layer"
        DS["Medical Image Data\n(Data Directory)"]:::dataModule
    end

    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        DS1["Dataset\n(utils/dataset.py)"]:::processingModule
        TH["Text Handler\n(utils/text_handler.py)"]:::processingModule
        VOC["Vocabulary\n(utils/vocabulary.py)"]:::processingModule
        PData["Preprocessed Data"]:::processingModule
    end

    %% Execution Layer
    subgraph "Execution Layer"
        DCP["Main Training Entry\n(dc.py)"]:::processingModule
        FD["DeepLearningFramework Decision"]:::processingModule
    end

    %% Shell Scripts
    subgraph "Shell Scripts"
        SM_CNN["Cross-modal CNN-RNN\n(cross_modal_cnn_rnn.sh)"]:::processingModule
        MM_CNN["Multi-modal CNN-RNN\n(multi_modal_cnn_rnn.sh)"]:::processingModule
        SM_KNN["Cross-modal k-NN\n(cross_modal_kNN.sh)"]:::processingModule
        MM_KNN["Multi-modal k-NN\n(multi_modal_kNN.sh)"]:::processingModule
    end

    %% Model & Processing Layer
    subgraph "Model & Processing Layer"
        subgraph "TensorFlow Models"
            crn["CNN-RNN\n(models/cnn_rnn.py)"]:::processingModule
            knn["k-NN\n(models/kNN.py)"]:::processingModule
            sota["SOTA Models"]:::processingModule
            sota1["R2Gen\n(SOTA_models/R2Gen)"]:::processingModule
            sota2["R2GenCMN\n(SOTA_models/R2GenCMN)"]:::processingModule
            sota3["VisualGPT\n(SOTA_models/VisualGPT)"]:::processingModule
        end
        pyt["ClinicalBERT\n(models/clinical_bert.py)"]:::processingModule
        subgraph "Inference/Decoding"
            beam["Beam Search\n(modules/beam_search.py)"]:::processingModule
            greedy["Greedy Search\n(modules/greedy_search.py)"]:::processingModule
            cluster["Cluster\n(modules/cluster.py)"]:::processingModule
            encoder["Image Encoder\n(modules/image_encoder.py)"]:::processingModule
        end
    end

    %% Evaluation Layer
    subgraph "Evaluation Layer"
        eval["Evaluation Utilities\n(utils/pycocoevalcap)"]:::evaluationModule
    end

    %% Data Flow Connections
    DS --> DS1
    DS --> TH
    DS --> VOC
    DS1 --> PData
    TH --> PData
    VOC --> PData
    PData --> DCP

    DCP --> FD
    FD -->|"TensorFlow"| crn
    FD -->|"TensorFlow"| knn
    FD -->|"TensorFlow"| sota
    FD -->|"PyTorch"| pyt

    SM_CNN --> crn
    MM_CNN --> crn
    SM_KNN --> knn
    MM_KNN --> knn

    crn --> beam
    knn --> beam
    sota --> beam
    pyt --> beam

    beam --> encoder
    beam --> cluster
    greedy --> eval
    encoder --> eval
    cluster --> eval
    beam -->|"outputs DiagnosticCaptions"| eval

    %% Click Events
    click DS "https://github.com/zaaachos/thesis-diagnostic-captioning/tree/main/data"
    click DS1 "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/utils/dataset.py"
    click TH "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/utils/text_handler.py"
    click VOC "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/utils/vocabulary.py"
    click DCP "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/dc.py"
    click SM_CNN "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/cross_modal_cnn_rnn.sh"
    click MM_CNN "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/multi_modal_cnn_rnn.sh"
    click SM_KNN "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/cross_modal_kNN.sh"
    click MM_KNN "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/multi_modal_kNN.sh"
    click crn "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/models/cnn_rnn.py"
    click knn "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/models/kNN.py"
    click pyt "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/models/clinical_bert.py"
    click sota1 "https://github.com/zaaachos/thesis-diagnostic-captioning/tree/main/SOTA_models/R2Gen"
    click sota2 "https://github.com/zaaachos/thesis-diagnostic-captioning/tree/main/SOTA_models/R2GenCMN"
    click sota3 "https://github.com/zaaachos/thesis-diagnostic-captioning/tree/main/SOTA_models/VisualGPT"
    click beam "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/modules/beam_search.py"
    click greedy "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/modules/greedy_search.py"
    click cluster "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/modules/cluster.py"
    click encoder "https://github.com/zaaachos/thesis-diagnostic-captioning/blob/main/modules/image_encoder.py"
    click eval "https://github.com/zaaachos/thesis-diagnostic-captioning/tree/main/utils/pycocoevalcap"

    %% Styles
    classDef dataModule fill:#e0f7fa,stroke:#006064,stroke-width:2px;
    classDef processingModule fill:#fff3e0,stroke:#e65100,stroke-width:2px;
    classDef evaluationModule fill:#f1f8e9,stroke:#33691e,stroke-width:2px;
```
