# junjue-wang/earthvqa architecture

> Architecture diagram of the GitHub repository junjue-wang/earthvqa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/junjue-wang/earthvqa
- Repository: https://github.com/junjue-wang/earthvqa

## Overview

##### Project Type & Purpose
- EarthVQA is a machine-learning pipeline (research codebase) for remote-sensing Visual Question Answering (VQA).
- It consists of two main sub-systems:
a. A semantic segmentation component (SFPNR50) to extract per-pixel land-cover features.
b. A relational-reasoning VQA module (SOBA) that consumes segmentation features plus question data to predict answers.
- The project supports both training and inference workflows: preparing data, training segmentation models, generating segmentation features (“pseudo-masks”), training the VQA model, and finally inference to answer queries.

##### File-Structure Analysis
- Top-level scripts & entry points:
- train_earthvqa.py, train_lovedav2_seg.py – train end-to-end or segmentation models on LoveDA and EarthVQA.
- predict_seg.py, predict_soba.py – inference scripts for segmentation feature generation and VQA answer generation.
- Directory breakdown:
- configs/ – YAML or Python config files for each model/dataset variant.
- data/ – data loader definitions for LoveDA and EarthVQA (splits, JSON QA files).
- module/ – neural network definitions:
– seg_base, semantic-fpn (segmentation backbone & decoder)
– soba, transformer, position_emb (VQA module components)
– loss.py, utils.py, viz.py
- utils/ – training utilities: loss functions, metrics computation.
- scripts/ – shell wrappers to chain training/inference steps.

##### Key Insights from README
- Two-stage pipeline:
1. Train segmentation model → generate segmentation features & pseudo-masks.
2. Train SOBA VQA model using image+mask features and question JSON → predict answers.
- Dependencies: PyTorch ≥1.1.0, ever-beta, segmentation_models.pytorch, albumentations (1.4.3).
- Data layout: separate Train / Val / Test directories with images_png, masks_png, and QA JSON files.

1. Data Store
- EarthVQA & LoveDA datasets (images, masks, QA JSON)
2. Configuration Manager
- configs/*.py feeding hyperparameters & paths into both pipelines
3. Segmentation Pipeline
- Data Loader (data/*.py) → SFPNR50 Model (module/seg_base.py + module/semantic-fpn.py) → Segmentation Features & Pseudo-Masks
4. VQA Pipeline
- Question Loader (data/*.py) + Image & Mask Features → SOBA Model (module/transformer.py, module/soba.py, module/position_emb.py) → Answer Prediction
5. Utilities & Metrics
- utils/loss.py, utils/metric.py for training loops in both pipelines
6. Orchestration Scripts
- scripts/*.sh or train_*.py / predict_*.py as glue
7. External Libraries
- PyTorch, ever-beta, segmentation_models.pytorch, albumentations

1. Data Store → (feeds) → Segmentation Pipeline
2. Config Manager → (parameterizes) → Segmentation & VQA Pipelines
3. Segmentation Pipeline → (outputs) → Segment-Feature Store
4. Segment-Feature Store + Data Store(QA JSON) → (feeds) → VQA Pipeline
5. VQA Pipeline → (outputs) → Answer Predictions
6. Both pipelines → (logging/metrics) → Utils & Metrics

c. Architectural Patterns & Design Principles
- Modular separation of concerns: Segmentation vs. VQA modules in distinct directories.
- Config-driven design: all hyperparameters and file paths abstracted into configs/*.py.
- Shell scripting for reproducible workflows.
- PyTorch-based pluggable models: easy to swap backbones or transformers.

d. Key Technologies & Frameworks
- Deep learning: PyTorch
- Segmentation: segmentation_models.pytorch, custom Semantic-FPN
- VQA: Transformer blocks, positional embeddings, relational reasoning via SOBA
- Data augmentation: albumentations
- Pipeline orchestration: bash scripts + Python entry points

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data & Configs"
        DS["Data Store (images, masks, QA JSON)"]:::dataConf
        CM["Configuration Manager (configs/)"]:::dataConf
    end
    click DS "https://github.com/junjue-wang/earthvqa/tree/master/data/"
    click CM "https://github.com/junjue-wang/earthvqa/tree/master/configs/"

    subgraph "Segmentation Subsystem"
        SDL1["LoveDA Loader"]:::seg
        SDL2["EarthVQA Loader"]:::seg
        SB["seg_base.py"]:::seg
        SP["semantic-fpn.py"]:::seg
        TLS1["train_lovedav2_seg.py"]:::seg
        TLS2["train_earthvqa.py"]:::seg
        TSH["train_sfpnr50.sh"]:::seg
        ISeg["predict_seg.py"]:::seg
        GSS["generate_segfeats.sh"]:::seg
        SFS["Segment-Feature Store<br/>(pseudo-masks & features)"]:::seg
    end
    click SDL1 "https://github.com/junjue-wang/earthvqa/blob/master/data/lovedav2.py"
    click SDL2 "https://github.com/junjue-wang/earthvqa/blob/master/data/earthvqa.py"
    click SB "https://github.com/junjue-wang/earthvqa/blob/master/module/seg_base.py"
    click SP "https://github.com/junjue-wang/earthvqa/blob/master/module/semantic-fpn.py"
    click TLS1 "https://github.com/junjue-wang/earthvqa/blob/master/train_lovedav2_seg.py"
    click TLS2 "https://github.com/junjue-wang/earthvqa/blob/master/train_earthvqa.py"
    click TSH "https://github.com/junjue-wang/earthvqa/blob/master/scripts/train_sfpnr50.sh"
    click ISeg "https://github.com/junjue-wang/earthvqa/blob/master/predict_seg.py"
    click GSS "https://github.com/junjue-wang/earthvqa/blob/master/scripts/generate_segfeats.sh"

    subgraph "VQA Subsystem"
        QL["Question Loader"]:::vqa
        SOB["soba.py"]:::vqa
        TRF["transformer.py"]:::vqa
        PEB["position_emb.py"]:::vqa
        TV1["train_earthvqa.py"]:::vqa
        TSH2["train_soba.sh"]:::vqa
        IVQA["predict_soba.py"]:::vqa
        PSSH["predict_soba.sh"]:::vqa
        AP["Answer Predictions"]:::vqa
    end
    click QL "https://github.com/junjue-wang/earthvqa/blob/master/data/earthvqa.py"
    click SOB "https://github.com/junjue-wang/earthvqa/blob/master/module/soba.py"
    click TRF "https://github.com/junjue-wang/earthvqa/blob/master/module/transformer.py"
    click PEB "https://github.com/junjue-wang/earthvqa/blob/master/module/position_emb.py"
    click TV1 "https://github.com/junjue-wang/earthvqa/blob/master/train_earthvqa.py"
    click TSH2 "https://github.com/junjue-wang/earthvqa/blob/master/scripts/train_soba.sh"
    click IVQA "https://github.com/junjue-wang/earthvqa/blob/master/predict_soba.py"
    click PSSH "https://github.com/junjue-wang/earthvqa/blob/master/scripts/predict_soba.sh"

    subgraph "Utilities & Metrics"
        UM1["module/utils.py"]:::utils
        UM2["module/loss.py"]:::utils
        UM3["module/viz.py"]:::utils
        UM4["utils/loss.py"]:::utils
        UM5["utils/metric.py"]:::utils
    end
    click UM1 "https://github.com/junjue-wang/earthvqa/blob/master/module/utils.py"
    click UM2 "https://github.com/junjue-wang/earthvqa/blob/master/module/loss.py"
    click UM3 "https://github.com/junjue-wang/earthvqa/blob/master/module/viz.py"
    click UM4 "https://github.com/junjue-wang/earthvqa/blob/master/utils/loss.py"
    click UM5 "https://github.com/junjue-wang/earthvqa/blob/master/utils/metric.py"

    OS["Orchestration Scripts (scripts/)"]:::utils
    click OS "https://github.com/junjue-wang/earthvqa/tree/master/scripts/"

    subgraph "External Libraries"
        EL1["PyTorch"]:::ext
        EL2["ever-beta"]:::ext
        EL3["segmentation_models.pytorch"]:::ext
        EL4["albumentations"]:::ext
    end

    DS --> SDL1
    DS --> SDL2
    CM --> SDL1
    CM --> SDL2
    CM --> SB
    CM --> SP
    SDL1 --> SB
    SDL2 --> SB
    SB --> SP
    SP --> SFS
    TLS1 --> SDL1
    TLS1 --> SB
    TLS1 --> SP
    TLS2 --> SDL2
    TLS2 --> SB
    TLS2 --> SP
    TSH --> SB
    TSH --> SP
    ISeg --> SFS
    GSS --> SFS

    SFS --> QL
    DS --> QL
    CM --> SOB
    CM --> TRF
    CM --> PEB
    QL --> SOB
    TRF --> SOB
    PEB --> SOB
    SOB --> AP
    TV1 --> QL
    TV1 --> SOB
    TV1 --> TRF
    TV1 --> PEB
    TSH2 --> QL
    TSH2 --> SOB
    TSH2 --> TRF
    TSH2 --> PEB
    IVQA --> AP
    PSSH --> AP

    SP --> UM1
    SP --> UM2
    SOB --> UM1
    SOB --> UM2
    SP --> UM3
    SOB --> UM3
    SP --> UM4
    SOB --> UM4
    SP --> UM5
    SOB --> UM5

    OS --> TLS1
    OS --> TLS2
    OS --> TSH
    OS --> ISeg
    OS --> GSS
    OS --> TV1
    OS --> TSH2
    OS --> IVQA
    OS --> PSSH

    classDef dataConf fill:#AED6F1,stroke:#1B4F72,color:#1B2631;
    classDef seg fill:#82E0AA,stroke:#196F3D,color:#0B5345;
    classDef vqa fill:#F5B041,stroke:#B9770E,color:#7E5109;
    classDef utils fill:#D5D8DC,stroke:#566573,color:#212F3C;
    classDef ext fill:#D6EAF8,stroke:#154360,color:#1B2631;
```
