How kaveeshasilva/pathvqa works
Project type and purpose
• PathVQA is an open-source research toolkit and dataset for medical Visual Question Answering (VQA).
• It provides:
• A large pathology VQA dataset (images + questions + answers)
• Two baseline methods (method1 and method2) with training, evaluation, and utilities
• An evaluation script for measuring performance
File structure
Top-level:
• Evaluation script/… – Python utilities for computing metrics on predictions
• baselines/… – Two separate baseline implementations
– method1/… (LXMERT-based pretrain, finetune, scripts, saved checkpoints)
– method2/… (custom VQA model, dataset builders, run scripts, tools)
Notice pipeline steps within each method:
– Data preparation (dataset builders, feature converters)
– Pretraining & fine-tuning entry points
– Run scripts (bash) to launch experiments
– Saved models and outputs
Config/build artifacts:
• requirements.txt files under each method
• .gitignore, .idea (IDE config)
README for additional insights
– Focused on building and evaluating VQA models on pathology images
– Links to paper, citation, dataset sources
a. Main components
1. Data Ingestion
• Raw pathology images + question-answer pairs (from textbooks/PEIR)
2. Data Preprocessing & Feature Extraction
• Method2 tools (detection_features_converter, adaptive converter)
• Tokenizers, dictionary creators
3. Model modules (two baselines)
• Baseline Method1 (LXMERT pipeline)
– Pretrain (lxmert_pretrain.py, pvqa.py)
– Finetune (entry.py, pvqa tasks)
• Baseline Method2 (custom VQA network)
– Pretrain_main.py, finetune_main.py, modeling.py
4. Training Orchestration
• Shell run scripts under baselines/method*/run/…
5. Model Checkpoint Storage
• baselines/method*/saved/…
6. Evaluation
• evaluation_script.py, utils.py
• Method-specific evaluate_main.py
7. Outputs & Metrics
• JSON answer files, performance scores
b. Relationships & data flow
• Raw Dataset → Preprocessing/Feature Extraction →
→ Model Pretraining → Model Fine-tuning →
→ Saved Checkpoints → Inference/Evaluation Script →
→ Final Metrics/JSON outputs
• Both Method1 and Method2 follow similar pipeline but differ in model internals and tools.
c. Architectural patterns & design principles
• Pipeline pattern: clear stages (ingestion → preprocess → train → evaluate)
• Modular organization: separate directories per baseline method, separate utils & tools
• Config-by-script: experiments launched via shell scripts for reproducibility
d. Key technologies/frameworks
• Python (PyTorch, NumPy, tokenizers)
• Shell scripting (bash) for experiment orchestration
• LXMERT codebase integrated into Method1
• Custom feature converters and VQA models in Method2