# kaveeshasilva/pathvqa architecture

> Architecture diagram of the GitHub repository kaveeshasilva/pathvqa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kaveeshasilva/pathvqa
- Repository: https://github.com/kaveeshasilva/pathvqa

## Overview

##### Project type and purpose
- PathVQA is an open-source research toolkit and dataset for medical Visual Question Answering (VQA).
- It provides:
- A large pathology VQA dataset (images + questions + answers)
- Two baseline methods (method1 and method2) with training, evaluation, and utilities
- An evaluation script for measuring performance

##### File structure
Top-level:
- Evaluation script/… – Python utilities for computing metrics on predictions
- baselines/… – Two separate baseline implementations
– method1/… (LXMERT-based pretrain, finetune, scripts, saved checkpoints)
– method2/… (custom VQA model, dataset builders, run scripts, tools)
Notice pipeline steps within each method:
– Data preparation (dataset builders, feature converters)
– Pretraining & fine-tuning entry points
– Run scripts (bash) to launch experiments
– Saved models and outputs
Config/build artifacts:
- requirements.txt files under each method
- .gitignore, .idea (IDE config)

##### README for additional insights
– Focused on building and evaluating VQA models on pathology images
– Links to paper, citation, dataset sources

a. Main components
1. Data Ingestion
- Raw pathology images + question-answer pairs (from textbooks/PEIR)
2. Data Preprocessing & Feature Extraction
- Method2 tools (detection_features_converter, adaptive converter)
- Tokenizers, dictionary creators
3. Model modules (two baselines)
- Baseline Method1 (LXMERT pipeline)
– Pretrain (lxmert_pretrain.py, pvqa.py)
– Finetune (entry.py, pvqa tasks)
- Baseline Method2 (custom VQA network)
– Pretrain_main.py, finetune_main.py, modeling.py
4. Training Orchestration
- Shell run scripts under baselines/method*/run/…
5. Model Checkpoint Storage
- baselines/method*/saved/…
6. Evaluation
- evaluation_script.py, utils.py
- Method-specific evaluate_main.py
7. Outputs & Metrics
- JSON answer files, performance scores
b. Relationships & data flow
- Raw Dataset → Preprocessing/Feature Extraction →
→ Model Pretraining → Model Fine-tuning →
→ Saved Checkpoints → Inference/Evaluation Script →
→ Final Metrics/JSON outputs
- Both Method1 and Method2 follow similar pipeline but differ in model internals and tools.
c. Architectural patterns & design principles
- Pipeline pattern: clear stages (ingestion → preprocess → train → evaluate)
- Modular organization: separate directories per baseline method, separate utils & tools
- Config-by-script: experiments launched via shell scripts for reproducibility
d. Key technologies/frameworks
- Python (PyTorch, NumPy, tokenizers)
- Shell scripting (bash) for experiment orchestration
- LXMERT codebase integrated into Method1
- Custom feature converters and VQA models in Method2

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Stage
    subgraph "Data Ingestion" 
        direction TB
        RawData["Raw Pathology Images + QA"]:::data
        click RawData "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/dataset.py"
    end

    subgraph "Data Preprocessing & Feature Extraction"
        direction TB
        DFC["detection_features_converter"]:::utilities
        click DFC "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/tools/detection_features_converter.py"
        ADFC["adaptive_detection_features_converter"]:::utilities
        click ADFC "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/tools/adaptive_detection_features_converter.py"
        CD["create_dictionary"]:::utilities
        click CD "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/tools/create_dictionary.py"
        CE["create_embedding"]:::utilities
        click CE "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/tools/create_embedding.py"
        VS["vqa-split"]:::utilities
        click VS "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/tools/vqa-split.py"
    end

    subgraph "Shared Utilities"
        direction TB
        Tok["tokenization.py"]:::utilities
        click Tok "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/lxrt/tokenization.py"
        FU["file_utils.py"]:::utilities
        click FU "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/lxrt/file_utils.py"
        MU["utils.py"]:::utilities
        click MU "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/utils.py"
        EvalU["utils.py (Eval)"]:::utilities
        click EvalU "https://github.com/kaveeshasilva/pathvqa/blob/main/Evaluation script/utils.py"
    end

    %% Model Modules
    subgraph "Model Modules" 
        direction TB
        subgraph "Method1 (LXMERT)" 
            direction TB
            subgraph "Pretraining" 
                LPT["lxmert_pretrain.py"]:::model
                click LPT "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/pretrain/lxmert_pretrain.py"
                PVQA_PT["pvqa.py"]:::model
                click PVQA_PT "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/pretrain/pvqa.py"
                LD["lxmert_data.py"]:::model
                click LD "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/pretrain/lxmert_data.py"
                QA["qa_answer_table.py"]:::model
                click QA "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/pretrain/qa_answer_table.py"
            end
            subgraph "Fine-tuning & PVQA Task" 
                EN["entry.py"]:::model
                click EN "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/entry.py"
                PVQA_FT["pvqa.py"]:::model
                click PVQA_FT "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/pvqa.py"
                PM["pvqa_model.py"]:::model
                click PM "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/tasks/pvqa_model.py"
                PD["pvqa_data.py"]:::model
                click PD "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/tasks/pvqa_data.py"
                PI["pvqa_ignore_ignoring.py"]:::model
                click PI "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/tasks/pvqa_ignore_ignoring.py"
            end
            subgraph "Model Definition & Optimization"
                M1["modeling.py"]:::model
                click M1 "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/lxrt/modeling.py"
                Opt1["optimization.py"]:::model
                click Opt1 "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/lxrt/optimization.py"
                P1["param.py"]:::model
                click P1 "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/src/param.py"
            end
        end
        subgraph "Method2 (Custom VQA)" 
            direction TB
            Pre2["pretrain_main.py"]:::model
            click Pre2 "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/pretrain_main.py"
            FT2["finetune_main.py"]:::model
            click FT2 "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/finetune_main.py"
            FTI2["finetune_main_ignore.py"]:::model
            click FTI2 "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/finetune_main_ignore.py"
            Eval2["evaluate_main.py"]:::model
            click Eval2 "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/evaluate_main.py"
            M2["modeling.py"]:::model
            click M2 "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/modeling.py"
            DS2["dataset.py"]:::model
            click DS2 "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/dataset.py"
            U2["utils.py"]:::model
            click U2 "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/utils.py"
        end
    end

    %% Orchestration
    subgraph "Training Orchestration"
        direction TB
        Run1["Method1 Run Scripts"]:::orchestration
        click Run1 "https://github.com/kaveeshasilva/pathvqa/tree/main/baselines/method1/run/"
        Run2["Method2 Run Scripts"]:::orchestration
        click Run2 "https://github.com/kaveeshasilva/pathvqa/tree/main/baselines/method2/run/"
    end

    %% Checkpoint Storage
    Check["Model Checkpoints (Method1)"]:::data
    click Check "https://github.com/kaveeshasilva/pathvqa/tree/main/baselines/method1/saved/lxmert/"

    %% Evaluation & Outputs
    subgraph "Evaluation"
        direction TB
        EvalScript["evaluation_script.py"]:::eval
        click EvalScript "https://github.com/kaveeshasilva/pathvqa/blob/main/Evaluation script/evaluation_script.py"
        MMU["utils.py"]:::eval
        click MMU "https://github.com/kaveeshasilva/pathvqa/blob/main/Evaluation script/utils.py"
        ME2["evaluate_main.py"]:::eval
        click ME2 "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method2/evaluate_main.py"
    end

    Outputs["Answer JSONs & Metrics"]:::eval
    click Outputs "https://github.com/kaveeshasilva/pathvqa/blob/main/baselines/method1/saved/lxmert/all_ans.json"

    %% Flow
    RawData --> DFC
    DFC --> ADFC --> CD --> CE --> VS
    VS --> Tok --> FU --> MU --> ModelModules
    ModelModules --> Run1
    ModelModules --> Run2
    Run1 --> Check
    Run2 --> Check
    Check --> EvalScript --> Outputs
    EvalScript --> ME2

    %% Styles
    classDef data fill:#ccf,stroke:#333,stroke-width:1px
    classDef model fill:#cfc,stroke:#333,stroke-width:1px
    classDef orchestration fill:#fc9,stroke:#333,stroke-width:1px
    classDef utilities fill:#ddd,stroke:#333,stroke-width:1px
    classDef eval fill:#f9c,stroke:#333,stroke-width:1px
```
