How thu-mig/yoloe works
Purpose and Entry Points
YOLOE is a PyTorch implementation of an open-vocabulary, real-time object detection and segmentation system. A single YOLOE model supports text prompts, visual prompts, and prompt-free recognition, then can be re-parameterized into conventional YOLOv8/YOLO11-compatible inference and transfer architectures.
Primary user entry points are task scripts: predict_text_prompt.py, predict_visual_prompt.py, predict_prompt_free.py, train_seg.py, train_vp.py, train_pe_free.py, val.py, and export.py. app.py hosts the optional local Gradio demo.
Framework Core and Dispatch Boundary
The vendored/extended Ultralytics package is the runtime API and lifecycle orchestrator. ultralytics/__init__.py exposes the public model interface; ultralytics/engine/model.py routes model operations; ultralytics/engine/predictor.py, ultralytics/engine/trainer.py, and ultralytics/engine/validator.py implement execution modes.
This is the principal abstraction boundary: scripts instantiate/configure a model, while engine classes own device execution, batch loops, results, callbacks, persistence, and mode-specific delegation.
Model Construction and YOLOE Specialization
Network definitions are declarative YAML configurations, notably ultralytics/cfg/models/v8/yoloe-v8-seg.yaml and ultralytics/cfg/models/11/yoloe-11-seg.yaml. They are parsed into PyTorch modules by ultralytics/nn/tasks.py, using generic layers and detection/segmentation heads in ultralytics/nn/modules/head.py.
YOLO task wrappers and standard task-specific loops live in ultralytics/models/yolo/model.py and its detect, segment, classify, pose, and obb subpackages. YOLOE-specific training, validation, and visual-prompt prediction extend that layer in ultralytics/models/yolo/yoloe/train_yoloe.py, ultralytics/models/yolo/yoloe/train_yoloe_seg.py, and ultralytics/models/yolo/yoloe/predict_vp.py.
Prompt Modalities and Re-parameterization
The architecture has three training/inference paths:
• Text prompts: region–text alignment training and detection/segmentation through train_seg.py and ultralytics/models/yolo/yoloe/train_seg.py.
• Visual prompts: trains the SAVPE visual prompt encoder separately via train_vp.py and ultralytics/models/yolo/yoloe/train_vp.py.
• Prompt-free: trains specialized vocabulary/prompt embeddings via train_pe_free.py and ultralytics/models/yolo/yoloe/train_pe.py.
After open-vocabulary pretraining, re-parameterization converts the model to the standard YOLO architecture for COCO linear probing/full tuning and export. Transfer launchers are train_pe.py and train_pe_all.py; model conversion/assembly utilities include tools/convert_segm2det.py, tools/get_vp_segm.py, and tools/get_pf_free_segm.py.
Data and Offline Asset Pipeline
The data layer loads, augments, converts, and batches YOLO/LVIS/grounding datasets through ultralytics/data/dataset.py, ultralytics/data/build.py, and ultralytics/data/augment.py. Dataset contracts reside in YAML, including ultralytics/cfg/datasets/lvis.yaml, ultralytics/cfg/datasets/Objects365v1.yaml, and ultralytics/cfg/datasets/coco.yaml.
Offline preprocessing is a material prerequisite for full training: SAM2 generates segmentation masks (tools/generate_sam_masks.py); grounding data and label embeddings are cached by tools/generate_grounding_cache.py and tools/generate_label_embedding.py. These produce local dataset-side artifacts rather than a managed data service.
Evaluation, Deployment, and External Dependencies
Validation scripts target LVIS zero-shot, prompt-free, visual-prompt, and COCO-transfer modes (val.py, val_vp.py, val_pe_free.py, val_coco.py); LVIS-specific metrics rely on vendored third_party/lvis-api/lvis/eval.py.
Runtime is Python/PyTorch, usually CUDA-capable; CPU, ARM, Jetson, Conda, Jupyter, and runner images are provided under docker/. Export flows through ultralytics/engine/exporter.py and export.py, targeting ONNX/CoreML and TensorRT benchmarking. MobileCLIP/CLIP provide text-embedding dependencies (third_party/ml-mobileclip/mobileclip/clip.py, third_party/CLIP/clip/clip.py); SAM2 is used for data-mask generation (third_party/sam2/sam2/build_sam.py). Pretrained weights and demos are retrieved or hosted through Hugging Face; the demo additionally requires Gradio/FastAPI.
Main entry points: train_yoloe.py YOLOE modes · model.py Operation dispatcher · predictor.py Execution loops · val.py Validation
Components 19
Entry points
- Task scripts · Prompt, training, validation, and export launchers.
- Gradio demo
app.py
Ultralytics runtime
- Public model API
ultralytics/__init__.py - Operation dispatcher
ultralytics/engine/model.py - Execution loops · Owns device execution, batch loops, callbacks, and results.
ultralytics/engine/predictor.py
YOLOE model
- Model YAMLs
ultralytics/cfg/models/v8/yoloe-v8-seg.yaml - Task builder
ultralytics/nn/tasks.py - Detection and seg heads
ultralytics/nn/modules/head.py - YOLO task layer
ultralytics/models/yolo/model.py - YOLOE modes · Text, visual-prompt, and prompt-free training and prediction.
ultralytics/models/yolo/yoloe/train_yoloe.py - Re-parameterization · Converts open-vocabulary models to conventional YOLO transfer architectures.
tools/convert_segm2det.py
Data and assets
- Dataset contracts
ultralytics/cfg/datasets/lvis.yaml - Data pipeline
ultralytics/data/dataset.py - Offline asset generation · Generates SAM masks, grounding caches, and label embeddings locally.
tools/generate_sam_masks.py - CLIP embeddings
third_party/ml-mobileclip/mobileclip/clip.py - SAM2
third_party/sam2/sam2/build_sam.py
Evaluation and deployment
- Validation · Covers LVIS zero-shot, visual/prompt-free, and COCO transfer modes.
val.py - LVIS metrics
third_party/lvis-api/lvis/eval.py - Model export · Exports ONNX/CoreML and supports TensorRT benchmarking.
ultralytics/engine/exporter.py
Connections 22
- Task scripts → Public model API: instantiates
- Gradio demo → Public model API: uses
- Public model API → Operation dispatcher: routes operations
- Operation dispatcher → Execution loops: delegates mode
- Operation dispatcher → YOLO task layer: selects task
- Model YAMLs → Task builder: defines network
- Task builder → Detection and seg heads: assembles
- YOLO task layer → YOLOE modes: extended by
- Execution loops → YOLOE modes: runs specialized modes
- YOLOE modes → Task builder: trains model
- Data pipeline → Execution loops: batches input
- Dataset contracts → Data pipeline: configures
- Offline asset generation → Data pipeline: produces training artifacts
- SAM2 → Offline asset generation: generates masks
- CLIP embeddings → YOLOE modes: supplies text embeddings
- YOLOE modes → Re-parameterization: converts after pretraining
- Re-parameterization → YOLO task layer: produces standard YOLO form
- Task scripts → Validation: launches
- Validation → Execution loops: uses validator loop
- Validation → LVIS metrics: uses for LVIS
- Task scripts → Model export: launches export
- Model export → Operation dispatcher: exports configured model