# shuyansy/earth-observation-vlms architecture

> Architecture diagram of the GitHub repository shuyansy/earth-observation-vlms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shuyansy/earth-observation-vlms
- Repository: https://github.com/shuyansy/earth-observation-vlms

## Overview

This repository contains two Earth-observation vision-language systems, EarthMind and TerraScope, with data handling, multimodal models, grounded segmentation, demos, and evaluation code. The principal workflow is a user providing Earth-observation imagery and a task, a model interpreting it and optionally producing grounded masks, and an answer or evaluation result. The README establishes the product family and its data, model, training, and evaluation scope; sampled source-index references support the internal model and dataset links shown. Other subsystem links are omitted where implementation wiring was not shown.

Main entry points: [`EarthMind/third_parts/sam2/modeling/sam2_base.py`](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam2_base.py) (Segmentation model), [`TerraScope/projects/terrascope/models/terrascope_model.py`](https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/models/terrascope_model.py) (TerraScope model), [`EarthMind/third_parts/sam2/modeling/sam2_utils.py`](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam2_utils.py) (SAM components), [`EarthMind/projects/llava_sam2/datasets/vqa_dataset.py`](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/llava_sam2/datasets/vqa_dataset.py) (Task datasets)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_earthmind["EarthMind"]
  node_earthmind_demo["EarthMind demo<br/>[demo.py]"]
  node_earthmind_data["Task datasets<br/>[vqa_dataset.py]"]
  node_earthmind_collate["Batch collation"]
  node_earthmind_helpers["Dataset utilities<br/>[utils.py]"]
  node_earthmind_model["EarthMind model<br/>[llava_sam2.py]"]
  node_earthmind_seg["Segmentation model<br/>[sam2_base.py]"]
  node_earthmind_video["Video tracking"]
  node_earthmind_ui["Interactive app<br/>[app.py]"]
end

subgraph group_terrascope["TerraScope"]
  node_terra_demo["TerraScope demo<br/>[demo.py]"]
  node_terra_data["Task datasets<br/>[GCG_Dataset.py]"]
  node_terra_model["TerraScope model"]
  node_terra_vision["Vision-language core<br/>[internvl.py]"]
  node_terra_grounding["Grounded reasoning<br/>[lisa.py]"]
  node_terra_preprocess["Image preprocessing<br/>[image_resize.py]"]
  node_terra_ui["Interactive app<br/>[app.py]"]
end

subgraph group_segmentation["SAM2 segmentation"]
  node_sam_prompts["Prompt encoder<br/>[prompt_encoder.py]"]
  node_sam_transformer["Mask transformer<br/>[transformer.py]"]
  node_sam_decoder["Mask decoder<br/>[mask_decoder.py]"]
  node_sam_utils["SAM components<br/>[sam2_utils.py]"]
  node_terra_video["Video tracking"]
end

subgraph group_evaluation["Evaluation"]
  node_earthmind_eval["EarthMind evaluation<br/>[run_evaluations.py]"]
  node_terra_eval["Benchmark evaluation"]
end

node_user(("EO user"))

node_user -.->|"provides task"| node_earthmind_demo
node_user -.->|"provides task"| node_terra_demo
node_earthmind_data -->|"uses"| node_earthmind_helpers
node_earthmind_data -->|"uses"| node_earthmind_collate
node_terra_model -->|"uses"| node_terra_vision
node_terra_model -->|"uses"| node_terra_grounding
node_terra_model -->|"uses"| node_terra_preprocess
node_earthmind_seg -->|"uses"| node_sam_prompts
node_earthmind_seg -->|"uses"| node_sam_transformer
node_earthmind_seg -->|"uses"| node_sam_decoder
node_sam_prompts -->|"uses"| node_sam_utils
node_sam_transformer -->|"uses"| node_sam_utils
node_sam_decoder -->|"uses"| node_sam_utils
node_earthmind_video -->|"uses"| node_earthmind_seg
node_terra_video -.->|"uses"| node_earthmind_seg

click node_earthmind_demo "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/demo.py"
click node_earthmind_data "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/llava_sam2/datasets/vqa_dataset.py"
click node_earthmind_collate "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/glamm/datasets/collate_fns/glamm_collate_fn.py"
click node_earthmind_helpers "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/glamm/datasets/utils/utils.py"
click node_earthmind_model "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/llava_sam2/models/llava_sam2.py"
click node_earthmind_seg "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam2_base.py"
click node_earthmind_video "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/sam2_video_predictor.py"
click node_earthmind_eval "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/llava_sam2/evaluation/run_evaluations.py"
click node_earthmind_ui "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/llava_sam2/gradio/app.py"
click node_terra_demo "https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/demo.py"
click node_terra_data "https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/datasets/GCG_Dataset.py"
click node_terra_model "https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/models/terrascope_model.py"
click node_terra_vision "https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/models/internvl.py"
click node_terra_grounding "https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/models/lisa.py"
click node_terra_preprocess "https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/models/preprocess/image_resize.py"
click node_terra_eval "https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/evaluation/eval_terrascope_bench.py"
click node_terra_ui "https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/gradio/app.py"
click node_sam_prompts "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam/prompt_encoder.py"
click node_sam_transformer "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam/transformer.py"
click node_sam_decoder "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam/mask_decoder.py"
click node_sam_utils "https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam2_utils.py"
click node_terra_video "https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/third_parts/sam2/sam2_video_predictor.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_earthmind_demo,node_earthmind_data,node_earthmind_collate,node_earthmind_helpers,node_earthmind_model,node_earthmind_seg,node_earthmind_video,node_earthmind_ui,node_user toneBlue
class node_terra_demo,node_terra_data,node_terra_model,node_terra_vision,node_terra_grounding,node_terra_preprocess,node_terra_ui toneAmber
class node_sam_prompts,node_sam_transformer,node_sam_decoder,node_sam_utils,node_terra_video toneMint
class node_earthmind_eval,node_terra_eval toneRose
```

## Components

### EarthMind

- [EarthMind demo](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/demo.py): `EarthMind/demo.py`
- [Task datasets](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/llava_sam2/datasets/vqa_dataset.py): `EarthMind/projects/llava_sam2/datasets/vqa_dataset.py`
- [Batch collation](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/glamm/datasets/collate_fns/glamm_collate_fn.py): `EarthMind/projects/glamm/datasets/collate_fns/glamm_collate_fn.py`
- [Dataset utilities](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/glamm/datasets/utils/utils.py): `EarthMind/projects/glamm/datasets/utils/utils.py`
- [EarthMind model](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/llava_sam2/models/llava_sam2.py): `EarthMind/projects/llava_sam2/models/llava_sam2.py`
- [Segmentation model](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam2_base.py): `EarthMind/third_parts/sam2/modeling/sam2_base.py`
- [Video tracking](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/sam2_video_predictor.py): `EarthMind/third_parts/sam2/sam2_video_predictor.py`
- [Interactive app](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/llava_sam2/gradio/app.py): `EarthMind/projects/llava_sam2/gradio/app.py`

### TerraScope

- [TerraScope demo](https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/demo.py): `TerraScope/demo.py`
- [Task datasets](https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/datasets/GCG_Dataset.py): `TerraScope/projects/terrascope/datasets/GCG_Dataset.py`
- [TerraScope model](https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/models/terrascope_model.py): `TerraScope/projects/terrascope/models/terrascope_model.py`
- [Vision-language core](https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/models/internvl.py): `TerraScope/projects/terrascope/models/internvl.py`
- [Grounded reasoning](https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/models/lisa.py): `TerraScope/projects/terrascope/models/lisa.py`
- [Image preprocessing](https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/models/preprocess/image_resize.py): `TerraScope/projects/terrascope/models/preprocess/image_resize.py`
- [Interactive app](https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/gradio/app.py): `TerraScope/projects/terrascope/gradio/app.py`

### SAM2 segmentation

- [Prompt encoder](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam/prompt_encoder.py): `EarthMind/third_parts/sam2/modeling/sam/prompt_encoder.py`
- [Mask transformer](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam/transformer.py): `EarthMind/third_parts/sam2/modeling/sam/transformer.py`
- [Mask decoder](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam/mask_decoder.py): `EarthMind/third_parts/sam2/modeling/sam/mask_decoder.py`
- [SAM components](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/third_parts/sam2/modeling/sam2_utils.py): `EarthMind/third_parts/sam2/modeling/sam2_utils.py`
- [Video tracking](https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/third_parts/sam2/sam2_video_predictor.py): `TerraScope/third_parts/sam2/sam2_video_predictor.py`

### Evaluation

- [EarthMind evaluation](https://github.com/shuyansy/earth-observation-vlms/blob/master/EarthMind/projects/llava_sam2/evaluation/run_evaluations.py): `EarthMind/projects/llava_sam2/evaluation/run_evaluations.py`
- [Benchmark evaluation](https://github.com/shuyansy/earth-observation-vlms/blob/master/TerraScope/projects/terrascope/evaluation/eval_terrascope_bench.py): `TerraScope/projects/terrascope/evaluation/eval_terrascope_bench.py`

### Other components

- **EO user**

## Connections

- EO user → EarthMind demo: provides task
- EO user → TerraScope demo: provides task
- Task datasets → Dataset utilities: uses (evidence: `EarthMind/projects/llava_sam2/datasets/vqa_dataset.py`)
- Task datasets → Batch collation: uses (evidence: `EarthMind/projects/llava_sam2/datasets/vqa_dataset.py`)
- TerraScope model → Vision-language core: uses (evidence: `TerraScope/projects/terrascope/models/terrascope_model.py`)
- TerraScope model → Grounded reasoning: uses (evidence: `TerraScope/projects/terrascope/models/terrascope_model.py`)
- TerraScope model → Image preprocessing: uses (evidence: `TerraScope/projects/terrascope/models/terrascope_model.py`)
- Segmentation model → Prompt encoder: uses (evidence: `EarthMind/third_parts/sam2/modeling/sam2_base.py`)
- Segmentation model → Mask transformer: uses (evidence: `EarthMind/third_parts/sam2/modeling/sam2_base.py`)
- Segmentation model → Mask decoder: uses (evidence: `EarthMind/third_parts/sam2/modeling/sam2_base.py`)
- Prompt encoder → SAM components: uses (evidence: `EarthMind/third_parts/sam2/modeling/sam/prompt_encoder.py`)
- Mask transformer → SAM components: uses (evidence: `EarthMind/third_parts/sam2/modeling/sam/transformer.py`)
- Mask decoder → SAM components: uses (evidence: `EarthMind/third_parts/sam2/modeling/sam/mask_decoder.py`)
- Video tracking → Segmentation model: uses (evidence: `EarthMind/third_parts/sam2/sam2_video_predictor.py`)
- Video tracking → Segmentation model: uses (evidence: `TerraScope/third_parts/sam2/sam2_video_predictor.py`)
