# renesas-rz/rzv_drp-ai_tvm architecture

> Architecture diagram of the GitHub repository renesas-rz/rzv_drp-ai_tvm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/renesas-rz/rzv_drp-ai_tvm
- Repository: https://github.com/renesas-rz/rzv_drp-ai_tvm

## Overview

1. **What this repository is**
   RUHMI for RZ/V is an end-to-end AI deployment stack for Renesas RZ/V series devices. Its core purpose is to take trained models from common ML formats, compile them through DRP-AI TVM/MERA, and run them on embedded boards using a device-side runtime.

2. **Top-level architecture split**
   The repo is organized around two major phases:
   - **Host-side model preparation and compilation**: converting, quantizing, pruning, and compiling models into runtime artifacts.
   - **Target-side inference execution**: C++ sample apps and runtime wrappers that load the compiled artifacts and execute inference on board hardware.

3. **Compiler and build toolchain**
   The `setup/`, `tvm/`, `3rdparty/`, `tutorials/`, and `obj/` areas form the build/compile layer. This is where TVM integration, MERA compiler support, runtime headers, and prebuilt runtime binaries live. The repository is set up to build a TVM-based compiler environment for the supported RZ/V targets.

4. **Model compilation pipeline**
   The `tutorials/` directory is the primary host-side entry point for model compilation. It contains scripts for ONNX, PyTorch, TFLite, and ExIR models, including quantized variants. These scripts drive the model import → optional preprocessing/quantization → DRP-AI compile flow and are the main place where model graphs are transformed into deployable runtime packages.

5. **Preprocessing and operator adaptation**
   `tutorials/drpai_preprocess/` contains the preprocessing logic that adapts models for DRP-AI constraints. `tutorials/exclude_operator/` shows another architectural boundary: models may need operator exclusions or graph adjustments before compilation to fit the accelerator/runtime capabilities.

6. **Conversion utilities and model acquisition**
   `convert/` contains helper scripts for downloading reference models and converting between formats such as PyTorch, ONNX, and TFLite. This layer is upstream of compilation: it prepares model inputs in a compiler-friendly format and serves as a bridge from external model ecosystems into the RUHMI pipeline.

7. **Runtime on the target board**
   `apps/` is the board-side inference layer. It contains C++ runtime wrappers (`MeraDrpRuntimeWrapper`, `PreRuntime`, `PreRuntimeV2H`) plus sample executables and toolchain files. This layer consumes the compiled model artifacts and handles inference execution on the embedded device, including board-specific variants for V2H versus the other RZ/V families.

8. **Application structure and boundary**
   The `apps/` code separates generic runtime handling from model/sample-specific logic. The wrappers abstract model loading and execution, while sample programs demonstrate classification, benchmarking, and image-based inference. The `apps/python/` directory provides a Python inference entry point and packaged wheels, showing that the runtime can also be accessed from Python on supported platforms.

9. **How-to sample applications**
   `how-to/sample_app/` and `how-to/sample_app_v2h/` are richer end-user demos that combine camera input, image conversion, model-specific postprocessing, and a websocket-based UI. These samples are architecturally important because they show the full runtime path: camera capture → preprocessing → inference → postprocessing → UI/telemetry output.

10. **Model-specific inference modules**
   Inside the how-to sample apps, the `recognize/` and `recognize_proc` layers encapsulate model adapters for tasks like classification, object detection, pose estimation, semantic segmentation, and face-related models. This is a key plug-in boundary: the app framework stays generic while per-model code handles input/output shapes and postprocessing.

11. **Pruning and model optimization**
   `pruning/` contains separate workflows for structured pruning and retraining, with examples for YOLOX, YOLOv8x, ResNet50, TopFormer, and ViT. This subsystem sits before compilation and is used to reduce model cost while keeping compatibility with the DRP-AI deployment path.

12. **Quantization-aware training**
   `QAT/` provides training scripts for PyTorch and TensorFlow. This is another upstream optimization path that feeds into the same compile/deploy pipeline, letting models be prepared for lower precision execution before they reach the compiler.

13. **Documentation as architecture map**
   `docs/` is not just reference material; it documents the supported model matrix, board-specific build paths, runtime wrappers, preprocessing behavior, and known errors. `how-to/` provides task-oriented examples and is effectively the integration layer for specific model families and deployment patterns.

14. **External and embedded dependencies**
   The repo vendors major dependencies such as TVM, protobuf, ONNX, DLPack, DMLC-core, and support libraries. These are not incidental: they define the compiler/runtime boundary, model IR support, and the embedded execution stack used to move from ML framework models to on-device inference.

15. **End-to-end data flow**
   The typical flow is: external model source → conversion or download helper → optional pruning/QAT/preprocessing → DRP-AI TVM compile → generated runtime binaries under `obj/` or app-specific assets → deployment to the RZ/V board → C++/Python runtime app executes inference on camera or file input → task-specific postprocessing and UI/output.

16. **Main architectural takeaway**
   This repo is a deployment platform, not a single application. Its core abstraction is the compiled model artifact plus a thin board runtime, with separate host-side tooling for preparation and board-side apps for execution, all organized around the DRP-AI TVM/MERA pipeline for Renesas RZ/V hardware.

Main entry points: [`tutorials`](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/tutorials) (Compile flows), [`apps/MeraDrpRuntimeWrapper.cpp`](https://github.com/renesas-rz/rzv_drp-ai_tvm/blob/main/apps/MeraDrpRuntimeWrapper.cpp) (Runtime wrap), [`tvm`](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/tvm) (Compiler stack), [`docs`](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/docs) (Docs)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_host["Host pipeline"]
  node_external_models["Model sources<br/>external ML models"]
  node_convert["Convert"]
  node_qat["QAT<br/>quantization training"]
  node_pruning["Pruning<br/>model slimming"]
  node_tutorials["Compile flows<br/>compile scripts"]
  node_preprocess["Preprocess<br/>DRP-AI adaptation"]
  node_exclude_ops["Exclude ops<br/>graph constraints"]
end

subgraph group_target["Target runtime"]
  node_runtime_wrap["Runtime wrap<br/>C++ wrapper"]
  node_pre_runtime["PreRuntime<br/>board runtime<br/>[PreRuntime.cpp]"]
  node_pre_runtime_v2h["PreRuntime V2H<br/>board runtime<br/>[PreRuntimeV2H.cpp]"]
  node_apps_cpp["C++ apps<br/>target samples"]
  node_apps_python["Python app<br/>python runtime"]
end

subgraph group_samples["Demos"]
  node_sample_app["Sample app<br/>camera/websocket demo"]
  node_sample_app_v2h["V2H demos<br/>camera demos"]
end

subgraph group_deps["Dependencies"]
  node_compiler_stack["Compiler stack<br/>TVM/MERA build"]
  node_setup_api["Runtime API<br/>C runtime headers"]
  node_third_party["3rd party<br/>vendored deps"]
  node_obj_artifacts[("Build artifacts<br/>compiled packages")]
end

node_docs["Docs"]

node_external_models -->|"fetch/convert"| node_convert
node_convert -->|"prepare"| node_qat
node_convert -->|"prepare"| node_pruning
node_external_models -->|"compile input"| node_tutorials
node_qat -->|"quantized model"| node_tutorials
node_pruning -->|"pruned model"| node_tutorials
node_tutorials -->|"adapt"| node_preprocess
node_tutorials -->|"constrain"| node_exclude_ops
node_preprocess -->|"compile"| node_compiler_stack
node_exclude_ops -->|"compile"| node_compiler_stack
node_compiler_stack -->|"emit"| node_obj_artifacts
node_setup_api -->|"support"| node_compiler_stack
node_third_party -->|"enable"| node_compiler_stack
node_obj_artifacts -->|"load"| node_runtime_wrap
node_runtime_wrap -->|"execute"| node_pre_runtime
node_runtime_wrap -->|"execute"| node_pre_runtime_v2h
node_pre_runtime -->|"used by"| node_apps_cpp
node_pre_runtime_v2h -->|"used by"| node_apps_cpp
node_apps_cpp -.->|"mirrored in"| node_apps_python
node_runtime_wrap -->|"drives"| node_sample_app
node_runtime_wrap -->|"drives"| node_sample_app_v2h
node_docs -.->|"documents"| node_tutorials
node_docs -.->|"documents"| node_apps_cpp
node_docs -.->|"documents"| node_sample_app
node_docs -.->|"documents"| node_sample_app_v2h

click node_convert "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/convert"
click node_qat "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/QAT"
click node_pruning "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/pruning"
click node_tutorials "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/tutorials"
click node_preprocess "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/tutorials/drpai_preprocess"
click node_exclude_ops "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/tutorials/exclude_operator"
click node_compiler_stack "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/tvm"
click node_setup_api "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/setup/include"
click node_third_party "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/3rdparty"
click node_obj_artifacts "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/obj"
click node_runtime_wrap "https://github.com/renesas-rz/rzv_drp-ai_tvm/blob/main/apps/MeraDrpRuntimeWrapper.cpp"
click node_pre_runtime "https://github.com/renesas-rz/rzv_drp-ai_tvm/blob/main/apps/PreRuntime.cpp"
click node_pre_runtime_v2h "https://github.com/renesas-rz/rzv_drp-ai_tvm/blob/main/apps/PreRuntimeV2H.cpp"
click node_apps_cpp "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/apps"
click node_apps_python "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/apps/python"
click node_sample_app "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/how-to/sample_app"
click node_sample_app_v2h "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/how-to/sample_app_v2h"
click node_docs "https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/docs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_external_models,node_convert,node_qat,node_pruning,node_tutorials,node_preprocess,node_exclude_ops toneBlue
class node_runtime_wrap,node_pre_runtime,node_pre_runtime_v2h,node_apps_cpp,node_apps_python toneAmber
class node_sample_app,node_sample_app_v2h toneMint
class node_compiler_stack,node_setup_api,node_third_party,node_obj_artifacts toneRose
class node_docs toneNeutral
```

## Components

### Host pipeline

Model preparation, conversion, optimization, and compilation on the development machine.

- **Model sources**: external ML models · Trained models from PyTorch, ONNX, TFLite, ExIR, and related ecosystems.
- [Convert](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/convert): model fetch/convert · Helpers that download reference models and translate them between framework formats. · `convert`
- [QAT](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/QAT): quantization training · PyTorch and TensorFlow quantization-aware training workflows. · `QAT`
- [Pruning](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/pruning): model slimming · Structured pruning and retraining flows that prepare models for deployment. · `pruning`
- [Compile flows](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/tutorials): compile scripts · Primary host-side entry point for compiling ONNX, PyTorch, TFLite, and ExIR models. · `tutorials`
- [Preprocess](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/tutorials/drpai_preprocess): DRP-AI adaptation · Pre-runtime preprocessing and graph adaptation for accelerator constraints. · `tutorials/drpai_preprocess`
- [Exclude ops](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/tutorials/exclude_operator): graph constraints · Operator exclusion and graph adjustment boundary before compilation. · `tutorials/exclude_operator`

### Target runtime

On-board inference runtimes, wrappers, and sample applications for RZ/V devices.

- [Runtime wrap](https://github.com/renesas-rz/rzv_drp-ai_tvm/blob/main/apps/MeraDrpRuntimeWrapper.cpp): C++ wrapper · Shared runtime wrapper around MERA/DRP-AI execution. · `apps/MeraDrpRuntimeWrapper.cpp`
- [PreRuntime](https://github.com/renesas-rz/rzv_drp-ai_tvm/blob/main/apps/PreRuntime.cpp): board runtime · Target-side preprocessing/runtime bridge for model execution. · `apps/PreRuntime.cpp`
- [PreRuntime V2H](https://github.com/renesas-rz/rzv_drp-ai_tvm/blob/main/apps/PreRuntimeV2H.cpp): board runtime · Variant of the target runtime tailored to V2H boards. · `apps/PreRuntimeV2H.cpp`
- [C++ apps](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/apps): target samples · Benchmarking and tutorial executables that load compiled artifacts and run inference. · `apps`
- [Python app](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/apps/python): python runtime · Python inference entry point and packaged wheels for supported targets. · `apps/python`

### Demos

Task-oriented example applications and integration samples.

- [Sample app](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/how-to/sample_app): camera/websocket demo · Generic camera-to-UI demo with model-specific recognize modules. · `how-to/sample_app`
- [V2H demos](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/how-to/sample_app_v2h): camera demos · Board-specific camera demos for classification, detection, depth, pose, and segmentation. · `how-to/sample_app_v2h`

### Dependencies

Vendored compiler/runtime dependencies and runtime artifacts.

- [Compiler stack](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/tvm): TVM/MERA build · TVM integration, compiler support, headers, and runtime build inputs. · `tvm`
- [Runtime API](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/setup/include): C runtime headers · Device and runtime API headers used by the compiler and target runtime. · `setup/include`
- [3rd party](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/3rdparty): vendored deps · Vendored libraries supporting IR, serialization, and utility code. · `3rdparty`
- [Build artifacts](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/obj): compiled packages · Prebuilt wheels and runtime binaries produced by the compilation toolchain. · `obj`

### Other components

- [Docs](https://github.com/renesas-rz/rzv_drp-ai_tvm/tree/main/docs): architecture docs · Model lists, compile APIs, runtime wrapper guidance, and conversion instructions. · `docs`

## Connections

- Model sources → Convert: fetch/convert · Reference models are acquired or translated into compiler-friendly formats.
- Convert → QAT: prepare · Converted models can feed quantization-aware training paths.
- Convert → Pruning: prepare · Converted models can feed pruning and retraining workflows.
- Model sources → Compile flows: compile input · Framework models enter the main compilation flows.
- QAT → Compile flows: quantized model · QAT outputs are compiled through the same deployment pipeline.
- Pruning → Compile flows: pruned model · Pruned/retrained models are compiled for deployment.
- Compile flows → Preprocess: adapt · Compilation flows invoke preprocessing and runtime adaptation steps.
- Compile flows → Exclude ops: constrain · Some models require operator exclusions before compilation.
- Preprocess → Compiler stack: compile · Preprocessed graphs are lowered through the TVM/MERA toolchain.
- Exclude ops → Compiler stack: compile · Adjusted graphs are lowered through the compiler stack.
- Compiler stack → Build artifacts: emit · Compiler produces runtime artifacts and wheels under obj/.
- Runtime API → Compiler stack: support · C runtime headers define the device/runtime boundary for compilation.
- 3rd party → Compiler stack: enable · Vendored libraries support compiler and model IR functionality.
- Build artifacts → Runtime wrap: load · Generated artifacts are consumed by the target-side wrapper.
- Runtime wrap → PreRuntime: execute · Generic runtime wrapper drives board inference execution.
- Runtime wrap → PreRuntime V2H: execute · V2H-specific runtime path uses the board-tailored pre-runtime.
- PreRuntime → C++ apps: used by · C++ applications call into the runtime layers for inference.
- PreRuntime V2H → C++ apps: used by · Board-specific apps use the V2H runtime path.
- C++ apps → Python app: mirrored in · The same deployment capability is exposed through Python on supported platforms.
- Runtime wrap → Sample app: drives · Generic sample app uses the runtime wrapper for camera-based demos.
- Runtime wrap → V2H demos: drives · V2H demos use the runtime wrapper and board-specific runtime path.
- Docs → Compile flows: documents · Docs explain the compile flows and supported model matrix.
- Docs → C++ apps: documents · Docs explain runtime wrappers and sample app usage.
- Docs → Sample app: documents · How-to material covers the generic camera/websocket demo stack.
- Docs → V2H demos: documents · How-to material covers board-specific demo applications.