jaychempan/lae-dino

Purpose and Top-Level Shape

LAE-DINO is a remote-sensing open-vocabulary object-detection project with two coupled subsystems: LAE-Label, an offline auto-annotation pipeline used to build LAE-1M, and a customized MMDetection fork that trains, evaluates, and serves LAE-DINO. It is a Python/CUDA research and batch-processing repository, not a web application.

Primary Entry Points

• Label generation starts from image directories via LAE-Label/det_with_sam.py, followed by ROI labeling through LAE-Label/internvl-infer.py or remote-compatible LAE-Label/internvl-infer-openai.py.

• Distributed training and evaluation are launched by mmdetection_lae/tools/dist_train.sh, mmdetection_lae/tools/dist_test.sh, backed by mmdetection_lae/tools/train.py and mmdetection_lae/tools/test.py.

• Image/directory inference is exposed through mmdetection_lae/demo/image_demo.py; programmatic inference APIs are in mmdetection_lae/mmdet/apis/det_inferencer.py.

Data Creation: LAE-Label Pipeline

1. Image preparation: large remote-sensing imagery can be tiled with LAE-Label/crop_huge_images.py.

2. Region proposal/segmentation: SAM produces regions of interest in LAE-Label/det_with_sam.py.

3. ROI extraction: masks/regions become cropped image inputs through LAE-Label/crop_rois_from_images.py.

4. Vision-language labeling: InternVL runs locally or against an OpenAI-style remote endpoint, assigning textual categories to crops. This is the boundary to external model weights, GPU inference, and optionally an OpenAI-compatible/lmdeploy service.

5. Annotation normalization and cleanup: detections convert to ODVG in LAE-Label/dets2odvg.py; ordered filtering, class extraction/merging, and NMS reside in LAE-Label/post_process/1_filter_annoatitions.py, LAE-Label/post_process/3_merge_classes.py, and LAE-Label/post_process/4_nms.py.

The resulting datasets are intended to be unified into COCO/ODVG-compatible annotations before detector training.

Training Data and Configuration Boundary

Experiment configurations are the control plane: they compose model, dataset, optimizer, runtime, and evaluation settings. LAE-specific pretraining and fine-tuning configurations are:

• mmdetection_lae/configs/lae_dino/lae_dino_swin-t_pretrain_LAE-1M.py

• mmdetection_lae/configs/lae_dino/lae_dino_swin-t_finetune_DIOR.py

• mmdetection_lae/configs/lae_dino/lae_dino_swin-t_finetune_DOTA.py

Dataset definitions and transforms enter the framework through mmdetection_lae/mmdet/datasets/odvg.py, mmdetection_lae/mmdet/datasets/base_det_dataset.py, and mmdetection_lae/mmdet/datasets/transforms/. Dataset artifacts themselves are external downloads (LAE-1M, DIOR, DOTA, etc.); the repository supplies configuration and conversion mechanisms rather than bundled data.

LAE-DINO Model Stack

The model is a DINO-derived open-vocabulary detector with Dynamic Vocabulary Construction and Visual-Guided Text Prompt Learning. Its LAE-specific detector and prediction head are concentrated in:

• mmdetection_lae/mmdet/models/detectors/lae_dino.py

• mmdetection_lae/mmdet/models/dense_heads/lae_dino_head.py

• mmdetection_lae/mmdet/models/language_models/bert.py

It builds on the broader registry-driven MMDetection architecture (mmdetection_lae/mmdet/registry.py): image preprocessing feeds a configurable backbone (including Swin), transformer/DINO detector layers, and detection head; text features from BERT participate in open-vocabulary matching. Core tensor/sample contracts are represented by mmdetection_lae/mmdet/structures/det_data_sample.py and bounding-box structures under mmdetection_lae/mmdet/structures/bbox/.

Execution, Evaluation, and Delivery

MMEngine/MMCV-style runners coordinate training loops, hooks, checkpointing, distributed execution, and evaluation through mmdetection_lae/mmdet/engine/runner/loops.py, mmdetection_lae/mmdet/engine/hooks/, and mmdetection_lae/mmdet/evaluation/metrics/coco_metric.py. Inference accepts user-supplied text classes and emits scored detections; visualization is handled by mmdetection_lae/mmdet/visualization/local_visualizer.py.

The material runtime is Python 3.8 with PyTorch/CUDA (documented around Torch 1.10/CUDA 11.3), MMEngine, MMCV, and MMDetection. External dependencies include SAM checkpoints, InternVL weights, BERT weights, Hugging Face/ModelScope-hosted artifacts, and optional TorchServe packaging via mmdetection_lae/tools/deployment/mmdet2torchserve.py.