# thu-mig/yoloe architecture

> Architecture diagram of the GitHub repository thu-mig/yoloe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thu-mig/yoloe
- Repository: https://github.com/thu-mig/yoloe

## Overview

#### Purpose and Entry Points
YOLOE is a PyTorch implementation of an open-vocabulary, real-time object detection and segmentation system. A single YOLOE model supports text prompts, visual prompts, and prompt-free recognition, then can be re-parameterized into conventional YOLOv8/YOLO11-compatible inference and transfer architectures.

Primary user entry points are task scripts: `predict_text_prompt.py`, `predict_visual_prompt.py`, `predict_prompt_free.py`, `train_seg.py`, `train_vp.py`, `train_pe_free.py`, `val.py`, and `export.py`. `app.py` hosts the optional local Gradio demo.

#### Framework Core and Dispatch Boundary
The vendored/extended Ultralytics package is the runtime API and lifecycle orchestrator. `ultralytics/__init__.py` exposes the public model interface; `ultralytics/engine/model.py` routes model operations; `ultralytics/engine/predictor.py`, `ultralytics/engine/trainer.py`, and `ultralytics/engine/validator.py` implement execution modes.

This is the principal abstraction boundary: scripts instantiate/configure a model, while engine classes own device execution, batch loops, results, callbacks, persistence, and mode-specific delegation.

#### Model Construction and YOLOE Specialization
Network definitions are declarative YAML configurations, notably `ultralytics/cfg/models/v8/yoloe-v8-seg.yaml` and `ultralytics/cfg/models/11/yoloe-11-seg.yaml`. They are parsed into PyTorch modules by `ultralytics/nn/tasks.py`, using generic layers and detection/segmentation heads in `ultralytics/nn/modules/head.py`.

YOLO task wrappers and standard task-specific loops live in `ultralytics/models/yolo/model.py` and its `detect`, `segment`, `classify`, `pose`, and `obb` subpackages. YOLOE-specific training, validation, and visual-prompt prediction extend that layer in `ultralytics/models/yolo/yoloe/train_yoloe.py`, `ultralytics/models/yolo/yoloe/train_yoloe_seg.py`, and `ultralytics/models/yolo/yoloe/predict_vp.py`.

#### Prompt Modalities and Re-parameterization
The architecture has three training/inference paths:
- **Text prompts:** region–text alignment training and detection/segmentation through `train_seg.py` and `ultralytics/models/yolo/yoloe/train_seg.py`.
- **Visual prompts:** trains the SAVPE visual prompt encoder separately via `train_vp.py` and `ultralytics/models/yolo/yoloe/train_vp.py`.
- **Prompt-free:** trains specialized vocabulary/prompt embeddings via `train_pe_free.py` and `ultralytics/models/yolo/yoloe/train_pe.py`.

After open-vocabulary pretraining, re-parameterization converts the model to the standard YOLO architecture for COCO linear probing/full tuning and export. Transfer launchers are `train_pe.py` and `train_pe_all.py`; model conversion/assembly utilities include `tools/convert_segm2det.py`, `tools/get_vp_segm.py`, and `tools/get_pf_free_segm.py`.

#### Data and Offline Asset Pipeline
The data layer loads, augments, converts, and batches YOLO/LVIS/grounding datasets through `ultralytics/data/dataset.py`, `ultralytics/data/build.py`, and `ultralytics/data/augment.py`. Dataset contracts reside in YAML, including `ultralytics/cfg/datasets/lvis.yaml`, `ultralytics/cfg/datasets/Objects365v1.yaml`, and `ultralytics/cfg/datasets/coco.yaml`.

Offline preprocessing is a material prerequisite for full training: SAM2 generates segmentation masks (`tools/generate_sam_masks.py`); grounding data and label embeddings are cached by `tools/generate_grounding_cache.py` and `tools/generate_label_embedding.py`. These produce local dataset-side artifacts rather than a managed data service.

#### Evaluation, Deployment, and External Dependencies
Validation scripts target LVIS zero-shot, prompt-free, visual-prompt, and COCO-transfer modes (`val.py`, `val_vp.py`, `val_pe_free.py`, `val_coco.py`); LVIS-specific metrics rely on vendored `third_party/lvis-api/lvis/eval.py`.

Runtime is Python/PyTorch, usually CUDA-capable; CPU, ARM, Jetson, Conda, Jupyter, and runner images are provided under `docker/`. Export flows through `ultralytics/engine/exporter.py` and `export.py`, targeting ONNX/CoreML and TensorRT benchmarking. MobileCLIP/CLIP provide text-embedding dependencies (`third_party/ml-mobileclip/mobileclip/clip.py`, `third_party/CLIP/clip/clip.py`); SAM2 is used for data-mask generation (`third_party/sam2/sam2/build_sam.py`). Pretrained weights and demos are retrieved or hosted through Hugging Face; the demo additionally requires Gradio/FastAPI.

Main entry points: [`ultralytics/models/yolo/yoloe/train_yoloe.py`](https://github.com/thu-mig/yoloe/blob/main/ultralytics/models/yolo/yoloe/train_yoloe.py) (YOLOE modes), [`ultralytics/engine/model.py`](https://github.com/thu-mig/yoloe/blob/main/ultralytics/engine/model.py) (Operation dispatcher), [`ultralytics/engine/predictor.py`](https://github.com/thu-mig/yoloe/blob/main/ultralytics/engine/predictor.py) (Execution loops), [`val.py`](https://github.com/thu-mig/yoloe/blob/main/val.py) (Validation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry points"]
  node_scripts["Task scripts<br/>CLI launchers"]
  node_demo["Gradio demo<br/>local demo<br/>[app.py]"]
end

subgraph group_runtime["Ultralytics runtime"]
  node_public_model["Public model API<br/>runtime API<br/>[__init__.py]"]
  node_engine_model["Operation dispatcher<br/>model lifecycle<br/>[model.py]"]
  node_execution_loops["Execution loops<br/>predict/train/validate<br/>[predictor.py]"]
end

subgraph group_model["YOLOE model"]
  node_yaml_configs["Model YAMLs<br/>network definitions<br/>[yoloe-v8-seg.yaml]"]
  node_task_builder["Task builder<br/>PyTorch assembly<br/>[tasks.py]"]
  node_heads["Detection and seg heads<br/>neural modules<br/>[head.py]"]
  node_yolo_tasks["YOLO task layer<br/>task wrappers<br/>[model.py]"]
  node_yoloe_modes["YOLOE modes<br/>specialized loops<br/>[train_yoloe.py]"]
  node_reparameterization["Re-parameterization<br/>transfer assembly"]
end

subgraph group_data["Data and assets"]
  node_dataset_contracts["Dataset contracts<br/>dataset YAML<br/>[lvis.yaml]"]
  node_data_pipeline["Data pipeline<br/>load/augment/batch<br/>[dataset.py]"]
  node_offline_assets["Offline asset generation<br/>preprocessing"]
  node_embedding_models["CLIP embeddings<br/>text encoder dependency<br/>[clip.py]"]
  node_sam2["SAM2<br/>mask generator dependency<br/>[build_sam.py]"]
end

subgraph group_delivery["Evaluation and deployment"]
  node_validation["Validation<br/>evaluation runner<br/>[val.py]"]
  node_lvis_metrics["LVIS metrics<br/>evaluation dependency<br/>[eval.py]"]
  node_exporter["Model export<br/>deployment exporter<br/>[exporter.py]"]
end

node_scripts -->|"instantiates"| node_public_model
node_demo -->|"uses"| node_public_model
node_public_model -->|"routes operations"| node_engine_model
node_engine_model -->|"delegates mode"| node_execution_loops
node_engine_model -->|"selects task"| node_yolo_tasks
node_yaml_configs -->|"defines network"| node_task_builder
node_task_builder -->|"assembles"| node_heads
node_yolo_tasks -->|"extended by"| node_yoloe_modes
node_execution_loops -->|"runs specialized modes"| node_yoloe_modes
node_yoloe_modes -->|"trains model"| node_task_builder
node_data_pipeline -->|"batches input"| node_execution_loops
node_dataset_contracts -->|"configures"| node_data_pipeline
node_offline_assets -->|"produces training artifacts"| node_data_pipeline
node_sam2 -->|"generates masks"| node_offline_assets
node_embedding_models -->|"supplies text embeddings"| node_yoloe_modes
node_yoloe_modes -->|"converts after pretraining"| node_reparameterization
node_reparameterization -->|"produces standard YOLO form"| node_yolo_tasks
node_scripts -->|"launches"| node_validation
node_validation -->|"uses validator loop"| node_execution_loops
node_validation -->|"uses for LVIS"| node_lvis_metrics
node_scripts -->|"launches export"| node_exporter
node_exporter -->|"exports configured model"| node_engine_model

click node_demo "https://github.com/thu-mig/yoloe/blob/main/app.py"
click node_public_model "https://github.com/thu-mig/yoloe/blob/main/ultralytics/__init__.py"
click node_engine_model "https://github.com/thu-mig/yoloe/blob/main/ultralytics/engine/model.py"
click node_execution_loops "https://github.com/thu-mig/yoloe/blob/main/ultralytics/engine/predictor.py"
click node_yaml_configs "https://github.com/thu-mig/yoloe/blob/main/ultralytics/cfg/models/v8/yoloe-v8-seg.yaml"
click node_task_builder "https://github.com/thu-mig/yoloe/blob/main/ultralytics/nn/tasks.py"
click node_heads "https://github.com/thu-mig/yoloe/blob/main/ultralytics/nn/modules/head.py"
click node_yolo_tasks "https://github.com/thu-mig/yoloe/blob/main/ultralytics/models/yolo/model.py"
click node_yoloe_modes "https://github.com/thu-mig/yoloe/blob/main/ultralytics/models/yolo/yoloe/train_yoloe.py"
click node_reparameterization "https://github.com/thu-mig/yoloe/blob/main/tools/convert_segm2det.py"
click node_dataset_contracts "https://github.com/thu-mig/yoloe/blob/main/ultralytics/cfg/datasets/lvis.yaml"
click node_data_pipeline "https://github.com/thu-mig/yoloe/blob/main/ultralytics/data/dataset.py"
click node_offline_assets "https://github.com/thu-mig/yoloe/blob/main/tools/generate_sam_masks.py"
click node_embedding_models "https://github.com/thu-mig/yoloe/blob/main/third_party/ml-mobileclip/mobileclip/clip.py"
click node_sam2 "https://github.com/thu-mig/yoloe/blob/main/third_party/sam2/sam2/build_sam.py"
click node_validation "https://github.com/thu-mig/yoloe/blob/main/val.py"
click node_lvis_metrics "https://github.com/thu-mig/yoloe/blob/main/third_party/lvis-api/lvis/eval.py"
click node_exporter "https://github.com/thu-mig/yoloe/blob/main/ultralytics/engine/exporter.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scripts,node_demo toneBlue
class node_public_model,node_engine_model,node_execution_loops toneAmber
class node_yaml_configs,node_task_builder,node_heads,node_yolo_tasks,node_yoloe_modes,node_reparameterization toneMint
class node_dataset_contracts,node_data_pipeline,node_offline_assets,node_embedding_models,node_sam2 toneRose
class node_validation,node_lvis_metrics,node_exporter toneIndigo
```

## Components

### Entry points

- **Task scripts**: CLI launchers · Prompt, training, validation, and export launchers.
- [Gradio demo](https://github.com/thu-mig/yoloe/blob/main/app.py): local demo · `app.py`

### Ultralytics runtime

- [Public model API](https://github.com/thu-mig/yoloe/blob/main/ultralytics/__init__.py): runtime API · `ultralytics/__init__.py`
- [Operation dispatcher](https://github.com/thu-mig/yoloe/blob/main/ultralytics/engine/model.py): model lifecycle · `ultralytics/engine/model.py`
- [Execution loops](https://github.com/thu-mig/yoloe/blob/main/ultralytics/engine/predictor.py): predict/train/validate · Owns device execution, batch loops, callbacks, and results. · `ultralytics/engine/predictor.py`

### YOLOE model

- [Model YAMLs](https://github.com/thu-mig/yoloe/blob/main/ultralytics/cfg/models/v8/yoloe-v8-seg.yaml): network definitions · `ultralytics/cfg/models/v8/yoloe-v8-seg.yaml`
- [Task builder](https://github.com/thu-mig/yoloe/blob/main/ultralytics/nn/tasks.py): PyTorch assembly · `ultralytics/nn/tasks.py`
- [Detection and seg heads](https://github.com/thu-mig/yoloe/blob/main/ultralytics/nn/modules/head.py): neural modules · `ultralytics/nn/modules/head.py`
- [YOLO task layer](https://github.com/thu-mig/yoloe/blob/main/ultralytics/models/yolo/model.py): task wrappers · `ultralytics/models/yolo/model.py`
- [YOLOE modes](https://github.com/thu-mig/yoloe/blob/main/ultralytics/models/yolo/yoloe/train_yoloe.py): specialized loops · Text, visual-prompt, and prompt-free training and prediction. · `ultralytics/models/yolo/yoloe/train_yoloe.py`
- [Re-parameterization](https://github.com/thu-mig/yoloe/blob/main/tools/convert_segm2det.py): transfer assembly · Converts open-vocabulary models to conventional YOLO transfer architectures. · `tools/convert_segm2det.py`

### Data and assets

- [Dataset contracts](https://github.com/thu-mig/yoloe/blob/main/ultralytics/cfg/datasets/lvis.yaml): dataset YAML · `ultralytics/cfg/datasets/lvis.yaml`
- [Data pipeline](https://github.com/thu-mig/yoloe/blob/main/ultralytics/data/dataset.py): load/augment/batch · `ultralytics/data/dataset.py`
- [Offline asset generation](https://github.com/thu-mig/yoloe/blob/main/tools/generate_sam_masks.py): preprocessing · Generates SAM masks, grounding caches, and label embeddings locally. · `tools/generate_sam_masks.py`
- [CLIP embeddings](https://github.com/thu-mig/yoloe/blob/main/third_party/ml-mobileclip/mobileclip/clip.py): text encoder dependency · `third_party/ml-mobileclip/mobileclip/clip.py`
- [SAM2](https://github.com/thu-mig/yoloe/blob/main/third_party/sam2/sam2/build_sam.py): mask generator dependency · `third_party/sam2/sam2/build_sam.py`

### Evaluation and deployment

- [Validation](https://github.com/thu-mig/yoloe/blob/main/val.py): evaluation runner · Covers LVIS zero-shot, visual/prompt-free, and COCO transfer modes. · `val.py`
- [LVIS metrics](https://github.com/thu-mig/yoloe/blob/main/third_party/lvis-api/lvis/eval.py): evaluation dependency · `third_party/lvis-api/lvis/eval.py`
- [Model export](https://github.com/thu-mig/yoloe/blob/main/ultralytics/engine/exporter.py): deployment exporter · Exports ONNX/CoreML and supports TensorRT benchmarking. · `ultralytics/engine/exporter.py`

## Connections

- Task scripts → Public model API: instantiates
- Gradio demo → Public model API: uses
- Public model API → Operation dispatcher: routes operations
- Operation dispatcher → Execution loops: delegates mode
- Operation dispatcher → YOLO task layer: selects task
- Model YAMLs → Task builder: defines network
- Task builder → Detection and seg heads: assembles
- YOLO task layer → YOLOE modes: extended by
- Execution loops → YOLOE modes: runs specialized modes
- YOLOE modes → Task builder: trains model
- Data pipeline → Execution loops: batches input
- Dataset contracts → Data pipeline: configures
- Offline asset generation → Data pipeline: produces training artifacts
- SAM2 → Offline asset generation: generates masks
- CLIP embeddings → YOLOE modes: supplies text embeddings
- YOLOE modes → Re-parameterization: converts after pretraining
- Re-parameterization → YOLO task layer: produces standard YOLO form
- Task scripts → Validation: launches
- Validation → Execution loops: uses validator loop
- Validation → LVIS metrics: uses for LVIS
- Task scripts → Model export: launches export
- Model export → Operation dispatcher: exports configured model
