# jiayuanwang-jw/yolov8-multi-task architecture

> Architecture diagram of the GitHub repository jiayuanwang-jw/yolov8-multi-task, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jiayuanwang-jw/yolov8-multi-task
- Repository: https://github.com/jiayuanwang-jw/yolov8-multi-task

## Overview

1. This repository is a fork of Ultralytics YOLOv8 adapted for the paper’s “A-YOLOM” multi-task setting. The core architectural goal is to run object detection plus multiple segmentation tasks in one shared model, while preserving the familiar YOLO training, validation, and prediction workflows.

2. The main runtime package is `ultralytics/`. That package contains the shared engine, model definitions, datasets, utilities, and task-specific entry points. Most of the architecture diagram should center on this package, with the rest of the repo acting as configuration, docs, examples, tests, and packaging.

3. The model boundary is defined by YAML model files under `ultralytics/models/` and by the network code in `ultralytics/nn/` and `ultralytics/yolo/`. The repo includes standard YOLOv8 variants, YOLOv3/YOLOv5 configs, RT-DETR and SAM-related modules, plus custom BDD-oriented YOLOv8 multi-task configs such as `yolov8-bdd*.yaml`. The custom multi-task behavior is wired into the YOLOv8 model graph rather than implemented as a separate application layer.

4. The task pipeline is split into mode-specific modules: `train`, `val`, `predict`, and `track`. Under `ultralytics/yolo/v8/` there are task-specific flows for detect, segment, classify, and pose; the repo also adds a multi-task path, including `DecSeg`, to support the paper’s combined detection/segmentation use case. These modes share the same engine abstractions, so the diagram should show one core engine branching into multiple task heads.

5. Data ingestion is managed by `ultralytics/yolo/data/`. This layer covers dataset construction, augmentation, dataloaders, converters, and stream loading. For the multi-task project, the key boundary is the dataset YAML plus a dataset directory that separates labels by task: detection-object, drivable segmentation, and lane segmentation. The dataset layer feeds task-aware batches into the trainer and predictor.

6. The multi-task data contract is important: the README describes a shared image set with parallel annotation trees, and the code expects the model YAML and dataset YAML to agree on total class count and per-task class counts. In other words, the architecture is not “one dataset, one label schema”; it is “one image source, multiple synchronized label sources” that are fused at training time.

7. The training/validation loop is centered in `ultralytics/yolo/engine/`. `trainer.py`, `validator.py`, `predictor.py`, `model.py`, `exporter.py`, and `results.py` define the shared orchestration around the model. This is where configuration from YAML, runtime arguments, callbacks, and device selection converge before being dispatched into the network and data subsystems.

8. The network internals live in `ultralytics/nn/` and lower-level building blocks under `ultralytics/nn/modules/`. These modules define the reusable convolutional, transformer, block, and head components that the model YAML composes. For diagramming, this is the backbone/neck/head area, with the multi-task heads branching from a shared feature extractor.

9. `ultralytics/yolo/utils/` provides cross-cutting support rather than a standalone subsystem: metrics, losses, ops, plotting, distributed execution, auto-batching, downloads, checks, and callbacks. The callback integrations are notable because they connect training runs to external experiment systems like TensorBoard, W&B, MLflow, Comet, Neptune, ClearML, and Ray Tune.

10. Tracking is a separate inference-time subsystem under `ultralytics/tracker/`. It includes ByteTrack and BoT-SORT implementations plus Kalman/matching/GMC utilities. In the architecture, tracking sits downstream of detection results and is optional, not part of training.

11. There are also two specialized vision branches: `ultralytics/vit/rtdetr/` and `ultralytics/vit/sam/`. These are separate model families for RT-DETR and SAM-style segmentation workflows, which indicates the repo is broader than the BDD multi-task paper code alone. They share the same general Ultralytics packaging and task interface, but are not the main path for the paper’s multi-task model.

12. The repo is packaged as a Python project with `setup.py`, `setup.cfg`, `requirements.txt`, and `MANIFEST.in`, and it includes Dockerfiles for CPU, ARM64, Jetson, and standard container builds. The checked-in `bin/`, `lib/`, and `pyvenv.cfg` indicate a bundled virtual environment in the tree, but architecturally the project is still a Python/PyTorch codebase targeting Python 3.7 and PyTorch 1.13.1.

13. Documentation and examples are substantial and should be treated as a separate support layer. `docs/` covers datasets, modes, models, callbacks, CLI/Python usage, and YOLOv5 compatibility/tutorials; `examples/` includes Python and C++ inference examples, notebooks, and ONNX/OpenCV integration. These are not runtime components, but they document the intended data flow and deployment surfaces.

14. Tests under `tests/` validate the command-line, engine, and Python APIs, which is useful for understanding stable boundaries. The tests imply the main public interfaces are the CLI, the `YOLO(...)` Python API, and the mode-specific engine behavior rather than any internal module directly.

15. For an architecture diagram, the clearest flow is: dataset YAML and labels -> data/augmentation/dataloaders -> engine orchestration -> shared backbone/neck -> task-specific heads -> results/metrics/callbacks, with optional branches for tracking, export, and external logging. The distinctive feature of this repo is that multiple tasks are fused into one model with shared features and task-specific outputs, rather than being separate models wired together externally.

Main entry points: [`ultralytics/yolo/engine/model.py`](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/model.py) (Engine), [`ultralytics/models/v8/yolov8-bdd.yaml`](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/models/v8/yolov8-bdd.yaml) (Model YAMLs), [`ultralytics/yolo/data/build.py`](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/data/build.py) (Data), [`ultralytics/yolo/v8/__init__.py`](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/v8/__init__.py) (Task flows)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Ultralytics runtime"]
  node_ultralytics_pkg["ultralytics<br/>python package<br/>[__init__.py]"]
  node_cfg["Config<br/>[default.yaml]"]
  node_engine["Engine<br/>orchestration<br/>[model.py]"]
  node_trainer["Trainer<br/>train loop<br/>[trainer.py]"]
  node_validator["Validator<br/>eval loop<br/>[validator.py]"]
  node_predictor["Predictor<br/>inference loop<br/>[predictor.py]"]
  node_results["Results<br/>outputs<br/>[results.py]"]
  node_data["Data<br/>ingest pipeline<br/>[build.py]"]
  node_datasets["Dataset YAMLs<br/>dataset specs<br/>[bdd-multi.yaml]"]
  node_tasks["Task flows<br/>mode entrypoints<br/>[__init__.py]"]
  node_decseg["DecSeg<br/>multi-task flow<br/>[train.py]"]
end

subgraph group_models["Model layer"]
  node_nn_tasks["NN tasks<br/>model factory<br/>[tasks.py]"]
  node_nn_modules["NN modules<br/>blocks<br/>[head.py]"]
  node_model_zoo["Model YAMLs<br/>model specs<br/>[yolov8-bdd.yaml]"]
end

subgraph group_support["Support layer"]
  node_tracker["Tracker<br/>online tracking<br/>[track.py]"]
  node_callbacks["Callbacks<br/>integrations<br/>[base.py]"]
  node_exporter["Exporter<br/>deployment<br/>[exporter.py]"]
  node_vit_rtdetr["RT-DETR<br/>vision branch<br/>[model.py]"]
  node_vit_sam["SAM<br/>vision branch<br/>[model.py]"]
  node_tests["Tests<br/>api checks<br/>[test_engine.py]"]
end

node_ultralytics_pkg -->|"reads"| node_cfg
node_ultralytics_pkg -->|"uses"| node_engine
node_engine -->|"dispatches train"| node_trainer
node_engine -->|"dispatches val"| node_validator
node_engine -->|"dispatches predict"| node_predictor
node_engine -->|"dispatches export"| node_exporter
node_engine -->|"returns"| node_results
node_trainer -->|"consumes"| node_data
node_validator -->|"consumes"| node_data
node_predictor -->|"prepares"| node_data
node_data -->|"configured by"| node_datasets
node_trainer -->|"hooks"| node_callbacks
node_validator -->|"hooks"| node_callbacks
node_predictor -->|"optional tracking"| node_tracker
node_tasks -->|"builds"| node_nn_tasks
node_nn_tasks -->|"composes"| node_nn_modules
node_nn_tasks -->|"parses"| node_model_zoo
node_model_zoo -->|"specializes"| node_decseg
node_tasks -->|"extends"| node_decseg
node_tasks -->|"shares"| node_engine
node_model_zoo -->|"includes"| node_vit_rtdetr
node_model_zoo -->|"includes"| node_vit_sam
node_tests -->|"covers"| node_engine
node_tests -->|"covers"| node_ultralytics_pkg

click node_ultralytics_pkg "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/__init__.py"
click node_cfg "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/cfg/default.yaml"
click node_engine "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/model.py"
click node_trainer "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/trainer.py"
click node_validator "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/validator.py"
click node_predictor "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/predictor.py"
click node_results "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/results.py"
click node_data "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/data/build.py"
click node_datasets "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/datasets/bdd-multi.yaml"
click node_tasks "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/v8/__init__.py"
click node_decseg "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/v8/DecSeg/train.py"
click node_nn_tasks "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/nn/tasks.py"
click node_nn_modules "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/nn/modules/head.py"
click node_model_zoo "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/models/v8/yolov8-bdd.yaml"
click node_tracker "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/tracker/track.py"
click node_callbacks "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/utils/callbacks/base.py"
click node_exporter "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/exporter.py"
click node_vit_rtdetr "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/vit/rtdetr/model.py"
click node_vit_sam "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/vit/sam/model.py"
click node_tests "https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/tests/test_engine.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ultralytics_pkg,node_cfg,node_engine,node_trainer,node_validator,node_predictor,node_results,node_data,node_datasets,node_tasks,node_decseg toneBlue
class node_nn_tasks,node_nn_modules,node_model_zoo toneAmber
class node_tracker,node_callbacks,node_exporter,node_vit_rtdetr,node_vit_sam,node_tests toneMint
```

## Components

### Ultralytics runtime

Core package that drives training, validation, prediction, export, and shared task logic.

- [ultralytics](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/__init__.py): python package · Top-level package exposing the unified YOLO API and multi-task extensions. · `ultralytics/__init__.py`
- [Config](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/cfg/default.yaml): runtime config · Default runtime settings and CLI/Python configuration surface. · `ultralytics/yolo/cfg/default.yaml`
- [Engine](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/model.py): orchestration · Shared training, validation, prediction, export, and result orchestration. · `ultralytics/yolo/engine/model.py`
- [Trainer](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/trainer.py): train loop · Builds training state, drives optimization, and coordinates callbacks. · `ultralytics/yolo/engine/trainer.py`
- [Validator](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/validator.py): eval loop · Runs evaluation and metric accumulation for task-specific outputs. · `ultralytics/yolo/engine/validator.py`
- [Predictor](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/predictor.py): inference loop · Handles inference-time preprocessing, forward passes, and postprocessing. · `ultralytics/yolo/engine/predictor.py`
- [Results](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/results.py): outputs · Standard result objects returned by prediction, validation, and tracking flows. · `ultralytics/yolo/engine/results.py`
- [Data](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/data/build.py): ingest pipeline · Dataset construction, augmentation, loaders, and dataset utilities. · `ultralytics/yolo/data/build.py`
- [Dataset YAMLs](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/datasets/bdd-multi.yaml): dataset specs · Task datasets, including the multi-task BDD contracts and standard YOLO datasets. · `ultralytics/datasets/bdd-multi.yaml`
- [Task flows](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/v8/__init__.py): mode entrypoints · Mode-specific train/val/predict implementations for detect, segment, classify, pose, and DecSeg. · `ultralytics/yolo/v8/__init__.py`
- [DecSeg](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/v8/DecSeg/train.py): multi-task flow · Custom BDD-oriented multi-task path for combined detection and segmentation. · `ultralytics/yolo/v8/DecSeg/train.py`

### Model layer

YAML-defined model families and the reusable network building blocks they compose.

- [NN tasks](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/nn/tasks.py): model factory · Builds model graphs from YAML and connects heads to the shared backbone/neck. · `ultralytics/nn/tasks.py`
- [NN modules](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/nn/modules/head.py): blocks · Reusable convolutional, transformer, block, head, and utility components. · `ultralytics/nn/modules/head.py`
- [Model YAMLs](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/models/v8/yolov8-bdd.yaml): model specs · YOLOv3, YOLOv5, YOLOv8, RT-DETR, SAM, and custom BDD multi-task model definitions. · `ultralytics/models/v8/yolov8-bdd.yaml`

### Support layer

Tracking, docs, examples, tests, and packaging around the core runtime.

- [Tracker](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/tracker/track.py): online tracking · Optional ByteTrack and BoT-SORT tracking on top of detection results. · `ultralytics/tracker/track.py`
- [Callbacks](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/utils/callbacks/base.py): integrations · Experiment and logging hooks for TensorBoard, W&B, MLflow, Comet, Neptune, ClearML, Ray Tune, and Hub. · `ultralytics/yolo/utils/callbacks/base.py`
- [Exporter](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/yolo/engine/exporter.py): deployment · Converts trained models to external runtimes and deployment formats. · `ultralytics/yolo/engine/exporter.py`
- [RT-DETR](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/vit/rtdetr/model.py): vision branch · Separate RT-DETR model family sharing the Ultralytics interface. · `ultralytics/vit/rtdetr/model.py`
- [SAM](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/ultralytics/vit/sam/model.py): vision branch · Separate SAM-style segmentation family with its own encoders, decoders, and prompts. · `ultralytics/vit/sam/model.py`
- [Tests](https://github.com/jiayuanwang-jw/yolov8-multi-task/blob/main/tests/test_engine.py): api checks · Validates CLI, engine, and Python-level public interfaces. · `tests/test_engine.py`

## Connections

- ultralytics → Config: reads · Loads runtime defaults and CLI/Python configuration.
- ultralytics → Engine: uses · Exposes the main YOLO execution engine.
- Engine → Trainer: dispatches train · Routes fit-style runs into the training loop.
- Engine → Validator: dispatches val · Routes evaluation into the validation loop.
- Engine → Predictor: dispatches predict · Routes inference into the predictor.
- Engine → Exporter: dispatches export · Sends trained models to export backends.
- Engine → Results: returns · Standardizes inference and evaluation outputs.
- Trainer → Data: consumes · Builds batches through dataset and loader pipeline.
- Validator → Data: consumes · Evaluates over the same data layer and task contracts.
- Predictor → Data: prepares · Uses loading and preprocessing for image, stream, and batch inputs.
- Data → Dataset YAMLs: configured by · Dataset YAMLs define paths, classes, and multi-task label structure.
- Trainer → Callbacks: hooks · Emits training events to logging and experiment systems.
- Validator → Callbacks: hooks · Emits evaluation events and metrics updates.
- Predictor → Tracker: optional tracking · Can hand detection outputs to online trackers.
- Task flows → NN tasks: builds · Mode-specific flows instantiate the underlying model graph.
- NN tasks → NN modules: composes · Assembles backbone, neck, and task heads from reusable blocks.
- NN tasks → Model YAMLs: parses · Consumes YAML model definitions to create networks.
- Model YAMLs → DecSeg: specializes · Custom BDD multi-task YAMLs wire shared features to multiple heads.
- Task flows → DecSeg: extends · Adds a multi-task path alongside detect, segment, classify, and pose.
- Task flows → Engine: shares · All task modes reuse the same engine abstractions.
- Model YAMLs → RT-DETR: includes · Repository also carries RT-DETR model definitions and runtime.
- Model YAMLs → SAM: includes · Repository also carries SAM-style model definitions and runtime.
- Tests → Engine: covers · Checks the stable engine and CLI/Python boundaries.
- Tests → ultralytics: covers · Exercises the public package entry points.
