How mbzuai-oryx/geopixel works
Primary Components
• Adaptive Image Divider – A module responsible for partitioning high-resolution RS images into local and global regions.
• Vision Encoder(s) – There are two “vision” paths:
• The main Vision Encoder (likely incorporated in modules under model/geopixel.py or through the “IXC” modules in model/IXC) that understands the overall image structure.
• A Grounding Vision Encoder, which is specifically geared toward fine-grained, pixel-level interpretation (evident in parts of model/sam2 which includes segmentation mechanisms).
• Large Language Model (LLM) – Used for processing natural language queries. (Note the references to InternLM-XComposer in the README and modules in model/IXC.)
• Pixel Decoder – The component that takes the outputs from the encoders and produces the final pixel-wise segmentation maps.
• Data & Annotation Pipeline – Though not part of the runtime modules, the documentation (docs/dataset.md, docs/finetune.md, etc.) and additional scripts (data_mix.py, merge_lora_weights_and_save_hf_model.py) indicate a robust dataset and annotation pipeline for RS-GCG.
File Tree to the Components
• Directories like “model/IXC” and “model/sam2” house key modules that implement the LLM and vision processing parts respectively.
• For example, “model/IXC/modeling_internlm_xcomposer2.py” and “model/IXC/tokenization_internlm2.py” point to components connected with the language model.
• In “model/sam2”, files like “automatic_mask_generator.py” and “build_sam.py”, as well as the deeper “modeling/sam” subdirectory, represent the segmentation (mask decoding and pixel decoding) part.
• The “docs” folder provides detailed guides on dataset handling, training, finetuning, and inference, which can guide the flow between the training/inference pipelines.
• Other scripts such as “train.py”, “train.sh”, and “finetune.sh” denote the orchestration of the training processes.
• Utility scripts (e.g., ixc_utils.py, utils.py) should be represented as supporting modules that manage tasks like preprocessing, configuration, and orchestration.
• Create distinct blocks for each major system module:
• Follow by a block for “Adaptive Image Divider”, which segments the input image.
• Converge these streams into a “Pixel Decoder” that produces segmentation masks or refined per-pixel outputs.
• Separate sections for “Data Preparation / Annotation Pipeline” and “Training / Finetuning Infrastructure”, referencing the dataset files, utility scripts, and the various shell scripts provided.
Architectural Patterns and Technologies
• Modular structure which separates the image processing, language understanding, and pixel decoding functionalities.
• Use of a large-scale pretrained language model (InternLM-XComposer) merged with specialized vision modules (SAM2 for segmentation and related image encoders).
• Dependency on CUDA, PyTorch (and torchvision), and flash-attention2 for high-resolution performance.
• Optionally, include integration with external services such as dataset hosting on HuggingFace, given the release announcements and links provided in the README.