How junjue-wang/earthvqa works
Project Type & Purpose
• EarthVQA is a machine-learning pipeline (research codebase) for remote-sensing Visual Question Answering (VQA).
• It consists of two main sub-systems:
a. A semantic segmentation component (SFPNR50) to extract per-pixel land-cover features.
b. A relational-reasoning VQA module (SOBA) that consumes segmentation features plus question data to predict answers.
• The project supports both training and inference workflows: preparing data, training segmentation models, generating segmentation features (“pseudo-masks”), training the VQA model, and finally inference to answer queries.
File-Structure Analysis
• Top-level scripts & entry points:
• train_earthvqa.py, train_lovedav2_seg.py – train end-to-end or segmentation models on LoveDA and EarthVQA.
• predict_seg.py, predict_soba.py – inference scripts for segmentation feature generation and VQA answer generation.
• Directory breakdown:
• configs/ – YAML or Python config files for each model/dataset variant.
• data/ – data loader definitions for LoveDA and EarthVQA (splits, JSON QA files).
• module/ – neural network definitions:
– seg_base, semantic-fpn (segmentation backbone & decoder)
– soba, transformer, position_emb (VQA module components)
– loss.py, utils.py, viz.py
• utils/ – training utilities: loss functions, metrics computation.
• scripts/ – shell wrappers to chain training/inference steps.
Key Insights from README
• Two-stage pipeline:
1. Train segmentation model → generate segmentation features & pseudo-masks.
2. Train SOBA VQA model using image+mask features and question JSON → predict answers.
• Dependencies: PyTorch ≥1.1.0, ever-beta, segmentation_models.pytorch, albumentations (1.4.3).
• Data layout: separate Train / Val / Test directories with images_png, masks_png, and QA JSON files.
1. Data Store
• EarthVQA & LoveDA datasets (images, masks, QA JSON)
2. Configuration Manager
• configs/*.py feeding hyperparameters & paths into both pipelines
3. Segmentation Pipeline
• Data Loader (data/*.py) → SFPNR50 Model (module/seg_base.py + module/semantic-fpn.py) → Segmentation Features & Pseudo-Masks
4. VQA Pipeline
• Question Loader (data/*.py) + Image & Mask Features → SOBA Model (module/transformer.py, module/soba.py, module/position_emb.py) → Answer Prediction
5. Utilities & Metrics
• utils/loss.py, utils/metric.py for training loops in both pipelines
6. Orchestration Scripts
• scripts/*.sh or train_*.py / predict_*.py as glue
7. External Libraries
• PyTorch, ever-beta, segmentation_models.pytorch, albumentations
1. Data Store → (feeds) → Segmentation Pipeline
2. Config Manager → (parameterizes) → Segmentation & VQA Pipelines
3. Segmentation Pipeline → (outputs) → Segment-Feature Store
4. Segment-Feature Store + Data Store(QA JSON) → (feeds) → VQA Pipeline
5. VQA Pipeline → (outputs) → Answer Predictions
6. Both pipelines → (logging/metrics) → Utils & Metrics
c. Architectural Patterns & Design Principles
• Modular separation of concerns: Segmentation vs. VQA modules in distinct directories.
• Config-driven design: all hyperparameters and file paths abstracted into configs/*.py.
• Shell scripting for reproducible workflows.
• PyTorch-based pluggable models: easy to swap backbones or transformers.
d. Key Technologies & Frameworks
• Deep learning: PyTorch
• Segmentation: segmentation_models.pytorch, custom Semantic-FPN
• VQA: Transformer blocks, positional embeddings, relational reasoning via SOBA
• Data augmentation: albumentations
• Pipeline orchestration: bash scripts + Python entry points