# mbzuai-oryx/geopixel architecture

> Architecture diagram of the GitHub repository mbzuai-oryx/geopixel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mbzuai-oryx/geopixel
- Repository: https://github.com/mbzuai-oryx/geopixel

## Overview

##### Primary Components
- Adaptive Image Divider – A module responsible for partitioning high-resolution RS images into local and global regions.
- Vision Encoder(s) – There are two “vision” paths:
- The main Vision Encoder (likely incorporated in modules under model/geopixel.py or through the “IXC” modules in model/IXC) that understands the overall image structure.
- A Grounding Vision Encoder, which is specifically geared toward fine-grained, pixel-level interpretation (evident in parts of model/sam2 which includes segmentation mechanisms).
- Large Language Model (LLM) – Used for processing natural language queries. (Note the references to InternLM-XComposer in the README and modules in model/IXC.)
- Pixel Decoder – The component that takes the outputs from the encoders and produces the final pixel-wise segmentation maps.
- Data & Annotation Pipeline – Though not part of the runtime modules, the documentation (docs/dataset.md, docs/finetune.md, etc.) and additional scripts (data_mix.py, merge_lora_weights_and_save_hf_model.py) indicate a robust dataset and annotation pipeline for RS-GCG.

##### File Tree to the Components
- Directories like “model/IXC” and “model/sam2” house key modules that implement the LLM and vision processing parts respectively.
- For example, “model/IXC/modeling_internlm_xcomposer2.py” and “model/IXC/tokenization_internlm2.py” point to components connected with the language model.
- In “model/sam2”, files like “automatic_mask_generator.py” and “build_sam.py”, as well as the deeper “modeling/sam” subdirectory, represent the segmentation (mask decoding and pixel decoding) part.
- The “docs” folder provides detailed guides on dataset handling, training, finetuning, and inference, which can guide the flow between the training/inference pipelines.
- Other scripts such as “train.py”, “train.sh”, and “finetune.sh” denote the orchestration of the training processes.
- Utility scripts (e.g., ixc_utils.py, utils.py) should be represented as supporting modules that manage tasks like preprocessing, configuration, and orchestration.

- Create distinct blocks for each major system module:
- Follow by a block for “Adaptive Image Divider”, which segments the input image.
- Converge these streams into a “Pixel Decoder” that produces segmentation masks or refined per-pixel outputs.
- Separate sections for “Data Preparation / Annotation Pipeline” and “Training / Finetuning Infrastructure”, referencing the dataset files, utility scripts, and the various shell scripts provided.

##### Architectural Patterns and Technologies
- Modular structure which separates the image processing, language understanding, and pixel decoding functionalities.
- Use of a large-scale pretrained language model (InternLM-XComposer) merged with specialized vision modules (SAM2 for segmentation and related image encoders).
- Dependency on CUDA, PyTorch (and torchvision), and flash-attention2 for high-resolution performance.
- Optionally, include integration with external services such as dataset hosting on HuggingFace, given the release announcements and links provided in the README.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Input Module"
        RS["\"Remote Sensing Image\""]:::input
        NL["\"Natural Language Query\""]:::input
    end

    RS -->|"provides image"| AID["\"Adaptive Image Divider\""]:::vision

    AID -->|"processes regions"| MVE["\"Main Vision Encoder\""]:::vision

    AID -->|"extracts region"| AMG["\"Automatic Mask Generator\""]:::vision
    AID -->|"extracts region"| BS["\"Build SAM\""]:::vision

    NL -->|"provides text"| LLMX["\"Modeling InternLM-XComposer2\""]:::language
    NL -->|"provides text"| LLM2["\"Modeling InternLM2\""]:::language
    NL -->|"provides text"| LLMT["\"Tokenization InternLM2\""]:::language

    MVE -->|"features to"| PD["\"Pixel Decoder\""]:::decoder
    AMG -->|"features to"| PD
    BS -->|"features to"| PD
    LLMX -->|"fuses text"| PD
    LLM2 -->|"fuses text"| PD
    LLMT -->|"fuses text"| PD

    PD -->|"outputs"| OUT["\"Segmentation Masks & Descriptions Output\""]:::output

    subgraph "Data & Annotation Pipeline"
        DG["\"Dataset Guide\""]:::data
        FG["\"Finetune Guide\""]:::data
        DMS["\"Data Mix Script\""]:::data
        MLS["\"Merge Lora Script\""]:::data
    end

    %% Connect Data Pipeline to Input as supporting resource
    DG --- RS
    FG --- RS
    DMS --- RS
    MLS --- RS

    %% Click Events
    click AID "https://github.com/mbzuai-oryx/geopixel/blob/main/model/geopixel.py"
    click MVE "https://github.com/mbzuai-oryx/geopixel/blob/main/model/geopixel.py"
    click AMG "https://github.com/mbzuai-oryx/geopixel/blob/main/model/sam2/automatic_mask_generator.py"
    click BS "https://github.com/mbzuai-oryx/geopixel/blob/main/model/sam2/build_sam.py"
    click PD "https://github.com/mbzuai-oryx/geopixel/blob/main/model/sam2/modeling/sam/mask_decoder.py"
    click LLMX "https://github.com/mbzuai-oryx/geopixel/blob/main/model/IXC/modeling_internlm_xcomposer2.py"
    click LLM2 "https://github.com/mbzuai-oryx/geopixel/blob/main/model/IXC/modeling_internlm2.py"
    click LLMT "https://github.com/mbzuai-oryx/geopixel/blob/main/model/IXC/tokenization_internlm2.py"
    click DG "https://github.com/mbzuai-oryx/geopixel/blob/main/docs/dataset.md"
    click FG "https://github.com/mbzuai-oryx/geopixel/blob/main/docs/finetune.md"
    click DMS "https://github.com/mbzuai-oryx/geopixel/blob/main/data_mix.py"
    click MLS "https://github.com/mbzuai-oryx/geopixel/blob/main/merge_lora_weights_and_save_hf_model.py"

    %% Styles
    classDef input fill:#f2f2f2,stroke:#333,stroke-width:2px;
    classDef vision fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef language fill:#d4edda,stroke:#333,stroke-width:2px;
    classDef decoder fill:#fff3cd,stroke:#333,stroke-width:2px;
    classDef output fill:#f8d7da,stroke:#333,stroke-width:2px;
    classDef data fill:#e2e3e5,stroke:#333,stroke-width:2px;
```
