# jaychempan/lae-dino architecture

> Architecture diagram of the GitHub repository jaychempan/lae-dino, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jaychempan/lae-dino
- Repository: https://github.com/jaychempan/lae-dino

## Overview

#### Purpose and Top-Level Shape
LAE-DINO is a remote-sensing open-vocabulary object-detection project with two coupled subsystems: **LAE-Label**, an offline auto-annotation pipeline used to build LAE-1M, and a customized **MMDetection** fork that trains, evaluates, and serves LAE-DINO. It is a Python/CUDA research and batch-processing repository, not a web application.

#### Primary Entry Points
- Label generation starts from image directories via `LAE-Label/det_with_sam.py`, followed by ROI labeling through `LAE-Label/internvl-infer.py` or remote-compatible `LAE-Label/internvl-infer-openai.py`.
- Distributed training and evaluation are launched by `mmdetection_lae/tools/dist_train.sh`, `mmdetection_lae/tools/dist_test.sh`, backed by `mmdetection_lae/tools/train.py` and `mmdetection_lae/tools/test.py`.
- Image/directory inference is exposed through `mmdetection_lae/demo/image_demo.py`; programmatic inference APIs are in `mmdetection_lae/mmdet/apis/det_inferencer.py`.

#### Data Creation: LAE-Label Pipeline
1. **Image preparation:** large remote-sensing imagery can be tiled with `LAE-Label/crop_huge_images.py`.
2. **Region proposal/segmentation:** SAM produces regions of interest in `LAE-Label/det_with_sam.py`.
3. **ROI extraction:** masks/regions become cropped image inputs through `LAE-Label/crop_rois_from_images.py`.
4. **Vision-language labeling:** InternVL runs locally or against an OpenAI-style remote endpoint, assigning textual categories to crops. This is the boundary to external model weights, GPU inference, and optionally an OpenAI-compatible/lmdeploy service.
5. **Annotation normalization and cleanup:** detections convert to ODVG in `LAE-Label/dets2odvg.py`; ordered filtering, class extraction/merging, and NMS reside in `LAE-Label/post_process/1_filter_annoatitions.py`, `LAE-Label/post_process/3_merge_classes.py`, and `LAE-Label/post_process/4_nms.py`.

The resulting datasets are intended to be unified into COCO/ODVG-compatible annotations before detector training.

#### Training Data and Configuration Boundary
Experiment configurations are the control plane: they compose model, dataset, optimizer, runtime, and evaluation settings. LAE-specific pretraining and fine-tuning configurations are:
- `mmdetection_lae/configs/lae_dino/lae_dino_swin-t_pretrain_LAE-1M.py`
- `mmdetection_lae/configs/lae_dino/lae_dino_swin-t_finetune_DIOR.py`
- `mmdetection_lae/configs/lae_dino/lae_dino_swin-t_finetune_DOTA.py`

Dataset definitions and transforms enter the framework through `mmdetection_lae/mmdet/datasets/odvg.py`, `mmdetection_lae/mmdet/datasets/base_det_dataset.py`, and `mmdetection_lae/mmdet/datasets/transforms/`. Dataset artifacts themselves are external downloads (LAE-1M, DIOR, DOTA, etc.); the repository supplies configuration and conversion mechanisms rather than bundled data.

#### LAE-DINO Model Stack
The model is a DINO-derived open-vocabulary detector with Dynamic Vocabulary Construction and Visual-Guided Text Prompt Learning. Its LAE-specific detector and prediction head are concentrated in:
- `mmdetection_lae/mmdet/models/detectors/lae_dino.py`
- `mmdetection_lae/mmdet/models/dense_heads/lae_dino_head.py`
- `mmdetection_lae/mmdet/models/language_models/bert.py`

It builds on the broader registry-driven MMDetection architecture (`mmdetection_lae/mmdet/registry.py`): image preprocessing feeds a configurable backbone (including Swin), transformer/DINO detector layers, and detection head; text features from BERT participate in open-vocabulary matching. Core tensor/sample contracts are represented by `mmdetection_lae/mmdet/structures/det_data_sample.py` and bounding-box structures under `mmdetection_lae/mmdet/structures/bbox/`.

#### Execution, Evaluation, and Delivery
MMEngine/MMCV-style runners coordinate training loops, hooks, checkpointing, distributed execution, and evaluation through `mmdetection_lae/mmdet/engine/runner/loops.py`, `mmdetection_lae/mmdet/engine/hooks/`, and `mmdetection_lae/mmdet/evaluation/metrics/coco_metric.py`. Inference accepts user-supplied text classes and emits scored detections; visualization is handled by `mmdetection_lae/mmdet/visualization/local_visualizer.py`.

The material runtime is Python 3.8 with PyTorch/CUDA (documented around Torch 1.10/CUDA 11.3), MMEngine, MMCV, and MMDetection. External dependencies include SAM checkpoints, InternVL weights, BERT weights, Hugging Face/ModelScope-hosted artifacts, and optional TorchServe packaging via `mmdetection_lae/tools/deployment/mmdet2torchserve.py`.

Main entry points: [`mmdetection_lae/mmdet/models/detectors/lae_dino.py`](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/models/detectors/lae_dino.py) (LAE-DINO detector), [`LAE-Label/crop_rois_from_images.py`](https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/crop_rois_from_images.py) (ROI extraction), [`mmdetection_lae/mmdet/engine/runner/loops.py`](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/engine/runner/loops.py) (Runner and hooks), [`LAE-Label/det_with_sam.py`](https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/det_with_sam.py) (SAM region proposals)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_label["LAE-Label pipeline"]
  node_images["Remote-sensing images<br/>input data"]
  node_tiling["Image tiling<br/>preparation"]
  node_sam["SAM region proposals<br/>segmentation<br/>[det_with_sam.py]"]
  node_roi_crops["ROI extraction<br/>crop generation"]
  node_internvl["InternVL labeling<br/>vision-language inference<br/>[internvl-infer.py]"]
  node_internvl_remote["Remote InternVL client<br/>remote labeling"]
  node_normalization["ODVG conversion<br/>annotation conversion<br/>[dets2odvg.py]"]
  node_cleanup["Annotation cleanup<br/>post-processing"]
end

subgraph group_detector["LAE-DINO framework"]
  node_configs["Experiment configs<br/>configuration"]
  node_data_pipeline["Dataset and transforms<br/>data pipeline<br/>[odvg.py]"]
  node_registry["MMDetection registry<br/>component registry<br/>[registry.py]"]
  node_lae_dino["LAE-DINO detector<br/>open-vocabulary detector<br/>[lae_dino.py]"]
  node_detection_head["LAE-DINO head<br/>prediction head<br/>[lae_dino_head.py]"]
  node_bert["BERT text encoder<br/>language model<br/>[bert.py]"]
  node_train_test["Distributed train and test<br/>execution entry points<br/>[dist_train.sh]"]
  node_runner["Runner and hooks<br/>training orchestration<br/>[loops.py]"]
  node_evaluation["COCO evaluation<br/>metric<br/>[coco_metric.py]"]
  node_inference["Inference API and demo<br/>serving interface<br/>[det_inferencer.py]"]
  node_visualizer["Local visualization<br/>output rendering"]
end

node_datasets[("External datasets<br/>dataset artifacts")]

node_images -.->|"large images"| node_tiling
node_images -->|"direct input"| node_sam
node_tiling -->|"tiles"| node_sam
node_sam -->|"masks and regions"| node_roi_crops
node_roi_crops -->|"crop inputs"| node_internvl
node_roi_crops -.->|"crop inputs"| node_internvl_remote
node_internvl -->|"labeled detections"| node_normalization
node_internvl_remote -->|"labeled detections"| node_normalization
node_normalization -->|"ODVG annotations"| node_cleanup
node_cleanup -->|"LAE-1M annotations"| node_datasets
node_datasets -->|"images and annotations"| node_data_pipeline
node_configs -->|"declares components"| node_registry
node_configs -->|"selects experiment"| node_train_test
node_data_pipeline -->|"detection samples"| node_lae_dino
node_registry -->|"builds"| node_lae_dino
node_bert -->|"text features"| node_lae_dino
node_lae_dino -->|"visual features"| node_detection_head
node_train_test -->|"launches"| node_runner
node_runner -->|"trains and evaluates"| node_lae_dino
node_runner -->|"reports metrics"| node_evaluation
node_inference -->|"runs prediction"| node_lae_dino
node_inference -->|"detections"| node_visualizer

click node_tiling "https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/crop_huge_images.py"
click node_sam "https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/det_with_sam.py"
click node_roi_crops "https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/crop_rois_from_images.py"
click node_internvl "https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/internvl-infer.py"
click node_internvl_remote "https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/internvl-infer-openai.py"
click node_normalization "https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/dets2odvg.py"
click node_cleanup "https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/post_process/1_filter_annoatitions.py"
click node_configs "https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/configs/lae_dino/lae_dino_swin-t_pretrain_LAE-1M.py"
click node_data_pipeline "https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/datasets/odvg.py"
click node_registry "https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/registry.py"
click node_lae_dino "https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/models/detectors/lae_dino.py"
click node_detection_head "https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/models/dense_heads/lae_dino_head.py"
click node_bert "https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/models/language_models/bert.py"
click node_train_test "https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/tools/dist_train.sh"
click node_runner "https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/engine/runner/loops.py"
click node_evaluation "https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/evaluation/metrics/coco_metric.py"
click node_inference "https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/apis/det_inferencer.py"
click node_visualizer "https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/visualization/local_visualizer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_images,node_tiling,node_sam,node_roi_crops,node_internvl,node_internvl_remote,node_normalization,node_cleanup toneBlue
class node_configs,node_data_pipeline,node_registry,node_lae_dino,node_detection_head,node_bert,node_train_test,node_runner,node_evaluation,node_inference,node_visualizer toneAmber
class node_datasets toneNeutral
```

## Components

### LAE-Label pipeline

Offline auto-annotation workflow.

- **Remote-sensing images**: input data
- [Image tiling](https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/crop_huge_images.py): preparation · `LAE-Label/crop_huge_images.py`
- [SAM region proposals](https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/det_with_sam.py): segmentation · `LAE-Label/det_with_sam.py`
- [ROI extraction](https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/crop_rois_from_images.py): crop generation · `LAE-Label/crop_rois_from_images.py`
- [InternVL labeling](https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/internvl-infer.py): vision-language inference · Runs locally or via an OpenAI-compatible endpoint. · `LAE-Label/internvl-infer.py`
- [Remote InternVL client](https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/internvl-infer-openai.py): remote labeling · `LAE-Label/internvl-infer-openai.py`
- [ODVG conversion](https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/dets2odvg.py): annotation conversion · `LAE-Label/dets2odvg.py`
- [Annotation cleanup](https://github.com/jaychempan/lae-dino/blob/main/LAE-Label/post_process/1_filter_annoatitions.py): post-processing · Filters annotations, merges classes, and applies NMS. · `LAE-Label/post_process/1_filter_annoatitions.py`

### LAE-DINO framework

Training, evaluation, and inference fork.

- [Experiment configs](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/configs/lae_dino/lae_dino_swin-t_pretrain_LAE-1M.py): configuration · Compose model, data, optimization, runtime, and evaluation. · `mmdetection_lae/configs/lae_dino/lae_dino_swin-t_pretrain_LAE-1M.py`
- [Dataset and transforms](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/datasets/odvg.py): data pipeline · Loads ODVG data and prepares detection samples. · `mmdetection_lae/mmdet/datasets/odvg.py`
- [MMDetection registry](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/registry.py): component registry · Builds configured framework components. · `mmdetection_lae/mmdet/registry.py`
- [LAE-DINO detector](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/models/detectors/lae_dino.py): open-vocabulary detector · DINO-derived detector with dynamic vocabulary construction. · `mmdetection_lae/mmdet/models/detectors/lae_dino.py`
- [LAE-DINO head](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/models/dense_heads/lae_dino_head.py): prediction head · `mmdetection_lae/mmdet/models/dense_heads/lae_dino_head.py`
- [BERT text encoder](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/models/language_models/bert.py): language model · Provides text features for open-vocabulary matching. · `mmdetection_lae/mmdet/models/language_models/bert.py`
- [Distributed train and test](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/tools/dist_train.sh): execution entry points · Shell launchers invoke framework training and evaluation scripts. · `mmdetection_lae/tools/dist_train.sh`
- [Runner and hooks](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/engine/runner/loops.py): training orchestration · Coordinates loops, checkpoints, hooks, and distributed execution. · `mmdetection_lae/mmdet/engine/runner/loops.py`
- [COCO evaluation](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/evaluation/metrics/coco_metric.py): metric · `mmdetection_lae/mmdet/evaluation/metrics/coco_metric.py`
- [Inference API and demo](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/apis/det_inferencer.py): serving interface · Accepts images and user-supplied text classes, returning scored detections. · `mmdetection_lae/mmdet/apis/det_inferencer.py`
- [Local visualization](https://github.com/jaychempan/lae-dino/blob/main/mmdetection_lae/mmdet/visualization/local_visualizer.py): output rendering · `mmdetection_lae/mmdet/visualization/local_visualizer.py`

### Other components

- **External datasets**: dataset artifacts · LAE-1M, DIOR, DOTA, and compatible annotations are downloaded externally.

## Connections

- Remote-sensing images → Image tiling: large images
- Remote-sensing images → SAM region proposals: direct input
- Image tiling → SAM region proposals: tiles
- SAM region proposals → ROI extraction: masks and regions
- ROI extraction → InternVL labeling: crop inputs
- ROI extraction → Remote InternVL client: crop inputs
- InternVL labeling → ODVG conversion: labeled detections
- Remote InternVL client → ODVG conversion: labeled detections
- ODVG conversion → Annotation cleanup: ODVG annotations
- Annotation cleanup → External datasets: LAE-1M annotations
- External datasets → Dataset and transforms: images and annotations
- Experiment configs → MMDetection registry: declares components
- Experiment configs → Distributed train and test: selects experiment
- Dataset and transforms → LAE-DINO detector: detection samples
- MMDetection registry → LAE-DINO detector: builds
- BERT text encoder → LAE-DINO detector: text features
- LAE-DINO detector → LAE-DINO head: visual features
- Distributed train and test → Runner and hooks: launches
- Runner and hooks → LAE-DINO detector: trains and evaluates
- Runner and hooks → COCO evaluation: reports metrics
- Inference API and demo → LAE-DINO detector: runs prediction
- Inference API and demo → Local visualization: detections