How mbzuai-oryx/geochat works
Project type and purpose
• GeoChat is an open-source machine-learning toolkit: a domain-specific vision-language system for remote-sensing images. It provides modules to build, train, evaluate, and serve a grounded Large Vision-Language Model (a fine-tuned LLaVA/Vicuna variant with LoRA adapters and a CLIP visual encoder).
File structure
Top-level directories:
• geochat/
– core Python package containing subpackages for model definition, multimodal encoders/projectors, training, evaluation, and serving
• docs/
– design and usage documentation
• scripts/
– shell scripts and Python helpers for pretraining, finetuning (LoRA, QLoRA, SQA), merging weights, DeepSpeed configs
• playground/
– example data and prompt files
• demo_images, images/
– assets for README illustrations
• geochat_demo.py
– quick-start script to run a demo
• pyproject.toml
– packaging and dependency configuration
README for additional insights
• Core architecture: CLIP vision tower → MLP projector → Vicuna-1.5 LLM
• Training: pretrain with feature alignment, then visual instruction tuning on RS dataset via DeepSpeed ZeRO-3 and LoRA
• Serving: CLI and Gradio web UI, model worker + controller pattern
• Evaluation: batch scripts for various RS tasks (grounding, VQA, scene classification)
Main components
A. Data & Configuration
• Remote-sensing image dataset (HuggingFace JSON + zipped images)
• Scripted dataset merge & preprocess (scripts/, playground/)
• Hyperparameter / DeepSpeed / LoRA config (scripts/*.json, scripts/*.sh)
B. Model Definition
• Visual Backbone: CLIP ViT-L/14 (geochat/model/multimodal_encoder)
• MLP Projector: vision→language embedding (geochat/model/multimodal_projector)
• Language Model: Vicuna-1.5 adapters & MPT variants (geochat/model/language_model & apply_delta)
• Consolidation & LoRA delta builder (builder.py, make_delta.py, consolidate.py)
C. Training Pipeline
• Pretraining stage: feature alignment (scripts/pretrain.sh → geochat/train/train.py & geochat/train/geochat_trainer.py)
• Visual instruction tuning stage: finetune LoRA (scripts/finetune_*.sh → DeepSpeed → geochat/train/)
• Custom layers & patches (flash-attn monkey-patch, memory trainers)
D. Evaluation Suite
• Batch evaluation scripts (geochat/eval/*.py) for grounding, referring, scene, VQA
• Evaluation orchestration (docs/Evaluation.md)
E. Serving & Demo
• Controller & CLI (geochat/serve/controller.py, cli.py)
• Model worker process (geochat/serve/model_worker.py)
• Web UI: Gradio trial & web server (gradio_trial.py, gradio_web_server.py)
• Demo entrypoint (geochat_demo.py)
Relationships and interactions
• Data flows from raw images & JSON instructions → preprocessing scripts → PyTorch Datasets → Training Pipeline → saves LoRA-tuned checkpoints
• Checkpoints feed both Evaluation Suite (batch scripts consume model API) and Serving layer
• Core model comprises a multimodal encoder + projector feeding into a language model
Architectural patterns & design principles
• Modular “builder” pattern: separate responsibilities for encoder, projector, LLM, delta adapters
• LoRA deltas isolate fine-tuning from base model weights
• DeepSpeed ZeRO for memory-efficient multi-GPU training
• Separation of concerns: core model code vs. training scripts vs. serving code vs. evaluation
• Extensibility points: new multimodal projector types, alternative LLM backbones, instruction datasets
Technologies, frameworks, and libraries
• PyTorch & Flash-Attention (via monkey-patch)
• HuggingFace Transformers & tokenizers
• CLIP vision tower (OpenAI CLIP-ViT)
• LoRA adapters & DeepSpeed ZeRO-3
• Gradio for web demo
• Shell / Python scripts for orchestration
Summary
• Key modules in geochat/ (model, train, eval, serve)
• How data and checkpoints flow between those modules
• External dependencies (datasets, pre-trained models)
• The two training phases (feature alignment + instruction tuning)
• Serving stack (Controller → Worker → Core model → Response)