# mbzuai-oryx/geochat architecture

> Architecture diagram of the GitHub repository mbzuai-oryx/geochat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mbzuai-oryx/geochat
- Repository: https://github.com/mbzuai-oryx/geochat

## Overview

##### Project type and purpose
- GeoChat is an open-source machine-learning toolkit: a domain-specific vision-language system for remote-sensing images. It provides modules to build, train, evaluate, and serve a grounded Large Vision-Language Model (a fine-tuned LLaVA/Vicuna variant with LoRA adapters and a CLIP visual encoder).

##### File structure
Top-level directories:
- geochat/
– core Python package containing subpackages for model definition, multimodal encoders/projectors, training, evaluation, and serving
- docs/
– design and usage documentation
- scripts/
– shell scripts and Python helpers for pretraining, finetuning (LoRA, QLoRA, SQA), merging weights, DeepSpeed configs
- playground/
– example data and prompt files
- demo_images, images/
– assets for README illustrations
- geochat_demo.py
– quick-start script to run a demo
- pyproject.toml
– packaging and dependency configuration

##### README for additional insights
- Core architecture: CLIP vision tower → MLP projector → Vicuna-1.5 LLM
- Training: pretrain with feature alignment, then visual instruction tuning on RS dataset via DeepSpeed ZeRO-3 and LoRA
- Serving: CLI and Gradio web UI, model worker + controller pattern
- Evaluation: batch scripts for various RS tasks (grounding, VQA, scene classification)

##### Main components
A. Data & Configuration
- Remote-sensing image dataset (HuggingFace JSON + zipped images)
- Scripted dataset merge & preprocess (scripts/, playground/)
- Hyperparameter / DeepSpeed / LoRA config (scripts/*.json, scripts/*.sh)
B. Model Definition
- Visual Backbone: CLIP ViT-L/14 (geochat/model/multimodal_encoder)
- MLP Projector: vision→language embedding (geochat/model/multimodal_projector)
- Language Model: Vicuna-1.5 adapters & MPT variants (geochat/model/language_model & apply_delta)
- Consolidation & LoRA delta builder (builder.py, make_delta.py, consolidate.py)
C. Training Pipeline
- Pretraining stage: feature alignment (scripts/pretrain.sh → geochat/train/train.py & geochat/train/geochat_trainer.py)
- Visual instruction tuning stage: finetune LoRA (scripts/finetune_*.sh → DeepSpeed → geochat/train/)
- Custom layers & patches (flash-attn monkey-patch, memory trainers)
D. Evaluation Suite
- Batch evaluation scripts (geochat/eval/*.py) for grounding, referring, scene, VQA
- Evaluation orchestration (docs/Evaluation.md)
E. Serving & Demo
- Controller & CLI (geochat/serve/controller.py, cli.py)
- Model worker process (geochat/serve/model_worker.py)
- Web UI: Gradio trial & web server (gradio_trial.py, gradio_web_server.py)
- Demo entrypoint (geochat_demo.py)

##### Relationships and interactions
- Data flows from raw images & JSON instructions → preprocessing scripts → PyTorch Datasets → Training Pipeline → saves LoRA-tuned checkpoints
- Checkpoints feed both Evaluation Suite (batch scripts consume model API) and Serving layer
- Core model comprises a multimodal encoder + projector feeding into a language model

##### Architectural patterns & design principles
- Modular “builder” pattern: separate responsibilities for encoder, projector, LLM, delta adapters
- LoRA deltas isolate fine-tuning from base model weights
- DeepSpeed ZeRO for memory-efficient multi-GPU training
- Separation of concerns: core model code vs. training scripts vs. serving code vs. evaluation
- Extensibility points: new multimodal projector types, alternative LLM backbones, instruction datasets

##### Technologies, frameworks, and libraries
- PyTorch & Flash-Attention (via monkey-patch)
- HuggingFace Transformers & tokenizers
- CLIP vision tower (OpenAI CLIP-ViT)
- LoRA adapters & DeepSpeed ZeRO-3
- Gradio for web demo
- Shell / Python scripts for orchestration

##### Summary
- Key modules in geochat/ (model, train, eval, serve)
- How data and checkpoints flow between those modules
- External dependencies (datasets, pre-trained models)
- The two training phases (feature alignment + instruction tuning)
- Serving stack (Controller → Worker → Core model → Response)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Dependencies
    subgraph "External Dependencies"
        HF["HuggingFace Datasets"]:::external
        CLIPRepo["Pre-trained CLIP ViT-L/14"]:::external
        VicunaRepo["Pre-trained Vicuna-1.5"]:::external
    end

    %% Data & Configuration
    subgraph "Data & Configuration" 
        direction TB
        Data["RS images & JSON dataset"]:::external
        Playground["playground/data"]:::config
        PretrainScript["scripts/pretrain.sh"]:::config
        FinetuneScript["scripts/finetune.sh"]:::config
        FinetuneLoRAScript["scripts/finetune_lora.sh"]:::config
        FinetuneQLORAScript["scripts/finetune_qlora.sh"]:::config
        FinetuneSQAScript["scripts/finetune_sqa.sh"]:::config
        DSConfig1["scripts/zero2.json"]:::config
        DSConfig2["scripts/zero3.json"]:::config
        DSConfig3["scripts/zero3_offload.json"]:::config
    end

    %% Model Definition
    subgraph "Model Definition" 
        direction TB
        subgraph "Visual Backbone (CLIP)" 
            CLIPEnc["clip_encoder.py"]:::core
            CLIPBuilder["builder.py"]:::core
        end
        subgraph "MLP Projector" 
            ProjBuilder["builder.py"]:::core
        end
        subgraph "Language Model" 
            LLama["geochat_llama.py"]:::core
            MPTFile["geochat_mpt.py"]:::core
            MPTDir["mpt/"]:::core
        end
        subgraph "Delta Builder" 
            ModelBuilder["builder.py"]:::core
            MakeDelta["make_delta.py"]:::core
            Consolidate["consolidate.py"]:::core
            ApplyDelta["apply_delta.py"]:::core
        end
    end

    %% Training Pipeline
    subgraph "Training Pipeline" 
        direction TB
        subgraph "Pretraining Stage" 
            PretrainScript --> TrainPy["geochat/train/train.py"]:::pipeline
            TrainPy --> Trainer["geochat/train/geochat_trainer.py"]:::pipeline
        end
        subgraph "Instruction Tuning Stage" 
            FinetuneScript
            FinetuneLoRAScript
            FinetuneQLORAScript
            FinetuneSQAScript --> TrainPy
            TrainPy --> Trainer
        end
        subgraph "Custom Layers & Patches"
            FlashPatch["llama_flash_attn_monkey_patch.py"]:::pipeline
            MemTrain["train_mem.py"]:::pipeline
        end
    end

    %% Checkpoint
    Checkpoint["LoRA-tuned Checkpoint"]:::data

    %% Evaluation Suite
    subgraph "Evaluation Suite" 
        direction TB
        EvalGround["batch_geochat_grounding.py"]:::pipeline
        EvalRef["batch_geochat_referring.py"]:::pipeline
        EvalScene["batch_geochat_scene.py"]:::pipeline
        EvalVQA["batch_geochat_vqa.py"]:::pipeline
    end

    %% Serving & Demo
    subgraph "Serving & Demo" 
        direction TB
        Controller["controller.py"]:::serving
        CLI["cli.py"]:::serving
        Worker["model_worker.py"]:::serving
        Register["register_worker.py"]:::serving
        WebTrial["gradio_trial.py"]:::serving
        WebServer["gradio_web_server.py"]:::serving
        DemoEntry["geochat_demo.py"]:::serving
    end

    %% Flows
    HF --> Data
    CLIPRepo --> CLIPEnc
    VicunaRepo --> LLama
    VicunaRepo --> MPTFile
    Data --> Playground
    Playground --> PretrainScript
    PretrainScript -->|"load_dataset()"| TrainPy
    DSConfig1 --> PretrainScript
    DSConfig2 --> FinetuneScript
    DSConfig3 --> FinetuneQLORAScript
    TrainPy -->|"save_delta()"| Checkpoint
    Checkpoint --> EvalGround
    Checkpoint --> EvalRef
    Checkpoint --> EvalScene
    Checkpoint --> EvalVQA
    Checkpoint --> Controller
    Controller --> Worker
    Worker -->|"inference()"| CLIPEnc
    Worker -->|"inference()"| LLama
    Worker -->|"apply_delta()"| ApplyDelta
    Worker --> DemoEntry
    WebTrial --> Worker
    WebServer --> Worker
    CLI --> Controller
    Register --> Worker

    %% Click events
    click Playground "https://github.com/mbzuai-oryx/geochat/tree/main/playground/data"
    click PretrainScript "https://github.com/mbzuai-oryx/geochat/blob/main/scripts/pretrain.sh"
    click FinetuneScript "https://github.com/mbzuai-oryx/geochat/blob/main/scripts/finetune.sh"
    click FinetuneLoRAScript "https://github.com/mbzuai-oryx/geochat/blob/main/scripts/finetune_lora.sh"
    click FinetuneQLORAScript "https://github.com/mbzuai-oryx/geochat/blob/main/scripts/finetune_qlora.sh"
    click FinetuneSQAScript "https://github.com/mbzuai-oryx/geochat/blob/main/scripts/finetune_sqa.sh"
    click DSConfig1 "https://github.com/mbzuai-oryx/geochat/blob/main/scripts/zero2.json"
    click DSConfig2 "https://github.com/mbzuai-oryx/geochat/blob/main/scripts/zero3.json"
    click DSConfig3 "https://github.com/mbzuai-oryx/geochat/blob/main/scripts/zero3_offload.json"
    click CLIPEnc "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/model/multimodal_encoder/clip_encoder.py"
    click CLIPBuilder "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/model/multimodal_encoder/builder.py"
    click ProjBuilder "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/model/multimodal_projector/builder.py"
    click LLama "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/model/language_model/geochat_llama.py"
    click MPTFile "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/model/language_model/geochat_mpt.py"
    click MPTDir "https://github.com/mbzuai-oryx/geochat/tree/main/geochat/model/language_model/mpt/"
    click ModelBuilder "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/model/builder.py"
    click MakeDelta "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/model/make_delta.py"
    click Consolidate "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/model/consolidate.py"
    click ApplyDelta "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/model/apply_delta.py"
    click FlashPatch "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/train/llama_flash_attn_monkey_patch.py"
    click MemTrain "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/train/train_mem.py"
    click EvalGround "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/eval/batch_geochat_grounding.py"
    click EvalRef "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/eval/batch_geochat_referring.py"
    click EvalScene "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/eval/batch_geochat_scene.py"
    click EvalVQA "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/eval/batch_geochat_vqa.py"
    click Controller "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/serve/controller.py"
    click CLI "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/serve/cli.py"
    click Worker "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/serve/model_worker.py"
    click Register "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/serve/register_worker.py"
    click WebTrial "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/serve/gradio_trial.py"
    click WebServer "https://github.com/mbzuai-oryx/geochat/blob/main/geochat/serve/gradio_web_server.py"
    click DemoEntry "https://github.com/mbzuai-oryx/geochat/blob/main/geochat_demo.py"

    %% Styles
    classDef core fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef pipeline fill:#ccffcc,stroke:#333,stroke-width:1px
    classDef serving fill:#ffddcc,stroke:#333,stroke-width:1px
    classDef external fill:#eeeeee,stroke:#333,stroke-width:1px
    classDef config fill:#ddffdd,stroke:#333,stroke-width:1px
    classDef data fill:#ffeedd,stroke:#333,stroke-width:1px
```
