How jiayuanwang-jw/yolov8-multi-task works
1. This repository is a fork of Ultralytics YOLOv8 adapted for the paper’s “A-YOLOM” multi-task setting. The core architectural goal is to run object detection plus multiple segmentation tasks in one shared model, while preserving the familiar YOLO training, validation, and prediction workflows.
2. The main runtime package is ultralytics/. That package contains the shared engine, model definitions, datasets, utilities, and task-specific entry points. Most of the architecture diagram should center on this package, with the rest of the repo acting as configuration, docs, examples, tests, and packaging.
3. The model boundary is defined by YAML model files under ultralytics/models/ and by the network code in ultralytics/nn/ and ultralytics/yolo/. The repo includes standard YOLOv8 variants, YOLOv3/YOLOv5 configs, RT-DETR and SAM-related modules, plus custom BDD-oriented YOLOv8 multi-task configs such as yolov8-bdd*.yaml. The custom multi-task behavior is wired into the YOLOv8 model graph rather than implemented as a separate application layer.
4. The task pipeline is split into mode-specific modules: train, val, predict, and track. Under ultralytics/yolo/v8/ there are task-specific flows for detect, segment, classify, and pose; the repo also adds a multi-task path, including DecSeg, to support the paper’s combined detection/segmentation use case. These modes share the same engine abstractions, so the diagram should show one core engine branching into multiple task heads.
5. Data ingestion is managed by ultralytics/yolo/data/. This layer covers dataset construction, augmentation, dataloaders, converters, and stream loading. For the multi-task project, the key boundary is the dataset YAML plus a dataset directory that separates labels by task: detection-object, drivable segmentation, and lane segmentation. The dataset layer feeds task-aware batches into the trainer and predictor.
6. The multi-task data contract is important: the README describes a shared image set with parallel annotation trees, and the code expects the model YAML and dataset YAML to agree on total class count and per-task class counts. In other words, the architecture is not “one dataset, one label schema”; it is “one image source, multiple synchronized label sources” that are fused at training time.
7. The training/validation loop is centered in ultralytics/yolo/engine/. trainer.py, validator.py, predictor.py, model.py, exporter.py, and results.py define the shared orchestration around the model. This is where configuration from YAML, runtime arguments, callbacks, and device selection converge before being dispatched into the network and data subsystems.
8. The network internals live in ultralytics/nn/ and lower-level building blocks under ultralytics/nn/modules/. These modules define the reusable convolutional, transformer, block, and head components that the model YAML composes. For diagramming, this is the backbone/neck/head area, with the multi-task heads branching from a shared feature extractor.
9. ultralytics/yolo/utils/ provides cross-cutting support rather than a standalone subsystem: metrics, losses, ops, plotting, distributed execution, auto-batching, downloads, checks, and callbacks. The callback integrations are notable because they connect training runs to external experiment systems like TensorBoard, W&B, MLflow, Comet, Neptune, ClearML, and Ray Tune.
10. Tracking is a separate inference-time subsystem under ultralytics/tracker/. It includes ByteTrack and BoT-SORT implementations plus Kalman/matching/GMC utilities. In the architecture, tracking sits downstream of detection results and is optional, not part of training.
11. There are also two specialized vision branches: ultralytics/vit/rtdetr/ and ultralytics/vit/sam/. These are separate model families for RT-DETR and SAM-style segmentation workflows, which indicates the repo is broader than the BDD multi-task paper code alone. They share the same general Ultralytics packaging and task interface, but are not the main path for the paper’s multi-task model.
12. The repo is packaged as a Python project with setup.py, setup.cfg, requirements.txt, and MANIFEST.in, and it includes Dockerfiles for CPU, ARM64, Jetson, and standard container builds. The checked-in bin/, lib/, and pyvenv.cfg indicate a bundled virtual environment in the tree, but architecturally the project is still a Python/PyTorch codebase targeting Python 3.7 and PyTorch 1.13.1.
13. Documentation and examples are substantial and should be treated as a separate support layer. docs/ covers datasets, modes, models, callbacks, CLI/Python usage, and YOLOv5 compatibility/tutorials; examples/ includes Python and C++ inference examples, notebooks, and ONNX/OpenCV integration. These are not runtime components, but they document the intended data flow and deployment surfaces.
14. Tests under tests/ validate the command-line, engine, and Python APIs, which is useful for understanding stable boundaries. The tests imply the main public interfaces are the CLI, the YOLO(...) Python API, and the mode-specific engine behavior rather than any internal module directly.
15. For an architecture diagram, the clearest flow is: dataset YAML and labels -> data/augmentation/dataloaders -> engine orchestration -> shared backbone/neck -> task-specific heads -> results/metrics/callbacks, with optional branches for tracking, export, and external logging. The distinctive feature of this repo is that multiple tasks are fused into one model with shared features and task-specific outputs, rather than being separate models wired together externally.
Main entry points: model.py Engine · yolov8-bdd.yaml Model YAMLs · build.py Data · __init__.py Task flows
Components 20
Ultralytics runtime
Core package that drives training, validation, prediction, export, and shared task logic.
- ultralytics · Top-level package exposing the unified YOLO API and multi-task extensions.
ultralytics/__init__.py - Config · Default runtime settings and CLI/Python configuration surface.
ultralytics/yolo/cfg/default.yaml - Engine · Shared training, validation, prediction, export, and result orchestration.
ultralytics/yolo/engine/model.py - Trainer · Builds training state, drives optimization, and coordinates callbacks.
ultralytics/yolo/engine/trainer.py - Validator · Runs evaluation and metric accumulation for task-specific outputs.
ultralytics/yolo/engine/validator.py - Predictor · Handles inference-time preprocessing, forward passes, and postprocessing.
ultralytics/yolo/engine/predictor.py - Results · Standard result objects returned by prediction, validation, and tracking flows.
ultralytics/yolo/engine/results.py - Data · Dataset construction, augmentation, loaders, and dataset utilities.
ultralytics/yolo/data/build.py - Dataset YAMLs · Task datasets, including the multi-task BDD contracts and standard YOLO datasets.
ultralytics/datasets/bdd-multi.yaml - Task flows · Mode-specific train/val/predict implementations for detect, segment, classify, pose, and DecSeg.
ultralytics/yolo/v8/__init__.py - DecSeg · Custom BDD-oriented multi-task path for combined detection and segmentation.
ultralytics/yolo/v8/DecSeg/train.py
Model layer
YAML-defined model families and the reusable network building blocks they compose.
- NN tasks · Builds model graphs from YAML and connects heads to the shared backbone/neck.
ultralytics/nn/tasks.py - NN modules · Reusable convolutional, transformer, block, head, and utility components.
ultralytics/nn/modules/head.py - Model YAMLs · YOLOv3, YOLOv5, YOLOv8, RT-DETR, SAM, and custom BDD multi-task model definitions.
ultralytics/models/v8/yolov8-bdd.yaml
Support layer
Tracking, docs, examples, tests, and packaging around the core runtime.
- Tracker · Optional ByteTrack and BoT-SORT tracking on top of detection results.
ultralytics/tracker/track.py - Callbacks · Experiment and logging hooks for TensorBoard, W&B, MLflow, Comet, Neptune, ClearML, Ray Tune, and Hub.
ultralytics/yolo/utils/callbacks/base.py - Exporter · Converts trained models to external runtimes and deployment formats.
ultralytics/yolo/engine/exporter.py - RT-DETR · Separate RT-DETR model family sharing the Ultralytics interface.
ultralytics/vit/rtdetr/model.py - SAM · Separate SAM-style segmentation family with its own encoders, decoders, and prompts.
ultralytics/vit/sam/model.py - Tests · Validates CLI, engine, and Python-level public interfaces.
tests/test_engine.py
Connections 24
- ultralytics → Config: reads · Loads runtime defaults and CLI/Python configuration.
- ultralytics → Engine: uses · Exposes the main YOLO execution engine.
- Engine → Trainer: dispatches train · Routes fit-style runs into the training loop.
- Engine → Validator: dispatches val · Routes evaluation into the validation loop.
- Engine → Predictor: dispatches predict · Routes inference into the predictor.
- Engine → Exporter: dispatches export · Sends trained models to export backends.
- Engine → Results: returns · Standardizes inference and evaluation outputs.
- Trainer → Data: consumes · Builds batches through dataset and loader pipeline.
- Validator → Data: consumes · Evaluates over the same data layer and task contracts.
- Predictor → Data: prepares · Uses loading and preprocessing for image, stream, and batch inputs.
- Data → Dataset YAMLs: configured by · Dataset YAMLs define paths, classes, and multi-task label structure.
- Trainer → Callbacks: hooks · Emits training events to logging and experiment systems.
- Validator → Callbacks: hooks · Emits evaluation events and metrics updates.
- Predictor → Tracker: optional tracking · Can hand detection outputs to online trackers.
- Task flows → NN tasks: builds · Mode-specific flows instantiate the underlying model graph.
- NN tasks → NN modules: composes · Assembles backbone, neck, and task heads from reusable blocks.
- NN tasks → Model YAMLs: parses · Consumes YAML model definitions to create networks.
- Model YAMLs → DecSeg: specializes · Custom BDD multi-task YAMLs wire shared features to multiple heads.
- Task flows → DecSeg: extends · Adds a multi-task path alongside detect, segment, classify, and pose.
- Task flows → Engine: shares · All task modes reuse the same engine abstractions.
- Model YAMLs → RT-DETR: includes · Repository also carries RT-DETR model definitions and runtime.
- Model YAMLs → SAM: includes · Repository also carries SAM-style model definitions and runtime.
- Tests → Engine: covers · Checks the stable engine and CLI/Python boundaries.
- Tests → ultralytics: covers · Exercises the public package entry points.