How visionxlab/crossearth works
Project Type and Purpose
• CrossEarth is a research‐oriented machine learning library/toolkit rather than a typical front-end/ back-end web application.
• Its purpose is to provide a geospatial vision foundation model for domain‐generalizable semantic segmentation of remote sensing imagery. It bundles:
• Model code (backbones, heads, segmentors)
• Dataset handling (rs_dataset, tools/dataset_converters)
• Configuration management (configs/)
• Training/inference scripts (tools/)
• Model conversion utilities (convert_models/)
• Benchmarks and evaluation metrics
File Structure Analysis
• Top-level packages:
• CrossEarth/ – core model and metric code
– models/backbones, heads, segmentors
– dg_metrics.py
• rs_dataset/ – dataset definitions for remote sensing sources
• configs/ – hierarchical experiment configurations organized by dataset and model variant, built on _base_
• tools/ – CLI scripts for training (train.py), testing/inference (test.py), weight conversion, dataset conversion
• convert_models/ – specialized scripts to convert pretrained Vision Foundation Models (e.g., dinov2) to the CrossEarth format
• benchmarks/ – benchmark descriptions and evaluation protocols
• requirements.txt – dependency list (PyTorch, mmsegmentation, mmcv, mmdet, xformers, etc.)
Key Insights from README
• Leverages MMEngine, MMCV, MMSegmentation, MMDet, PyTorch 2.0, xFormers.
• Workflow: download pretrained weights → adjust config files (datasets + model settings) → python tools/test.py / tools/train.py.
• Domain generalization: many dataset cross‐domain experiments configured under configs/_base_/datasets.
• Outputs: segmentation masks, pseudo labels, checkpoints stored in CrossEarth/checkpoints.
a. Main Components
• Configuration Manager
• Reads experiment settings from configs/ (dataset paths, model parameters, training schedule).
• Data Pipeline
• rs_dataset module + tools/dataset_converters → dataset loader, augmentation, sampling.
• Model Module
• Backbones (e.g., dinov2, reins, vit variants)
• Heads (rein_mask2former)
• Segmentors (peft_encoder_decoder)
• Optimizer & Scheduler
• PEFT optimizer constructor in CrossEarth/optimizers + schedule_40k.py
• Training & Inference Engine
• tools/train.py orchestrates forward/backward passes, logging, checkpointing
• tools/test.py runs evaluation / pseudo label generation
• Conversion Utilities
• convert_models/*.py to adapt external pretrained weights
• Benchmarks & Metrics
• benchmarks/ + CrossEarth/dg_metrics.py for standardized evaluation
b. Component Interactions & Data Flow
1. User invokes CLI (train.py or test.py)
2. Config Manager loads experiment config → instantiates Dataset and Model
3. Data Pipeline reads raw imagery via rs_dataset → batches fed to Model
4. Model (Backbone → Head → Segmentor) outputs segmentation logits
5. In training:
• Optimizer updates model parameters
• Scheduler adjusts learning rate
• Checkpoints saved to disk
6. In inference:
• Model checkpoints loaded
• Predictions post-processed and saved as masks / pseudo labels
7. Benchmark scripts use dg_metrics.py to compute performance
c. Architectural Patterns & Design Principles
• Modular design: clear separation of dataset, model, optimizer, config, and CLI layers
• Plugin architecture (via MMEngine/MMCV): models and optimizers registered dynamically
• Configuration‐driven experiments: promotes reproducibility and easy domain-generalization experiments
• Conversion layer isolates weight adaptation from core training code
d. Key Technologies & Frameworks
• PyTorch 2.0 / CUDA 11.7
• OpenMIM / MMEngine / MMCV / MMSegmentation / MMDetection
• xFormers for efficient transformer backbones
• Hydra‐style / config‐file management in configs/
Diagramming Conventions
• Components as rectangles labeled clearly (e.g., “Backbone Module”, “Dataset Loader”)
• Blue for core modules (CrossEarth/)
• Green for data (rs_dataset, tools/dataset_converters)
• Orange for configs
• Purple for external services/dependencies