renesas-rz/rzv_drp-ai_tvm

1. What this repository is

RUHMI for RZ/V is an end-to-end AI deployment stack for Renesas RZ/V series devices. Its core purpose is to take trained models from common ML formats, compile them through DRP-AI TVM/MERA, and run them on embedded boards using a device-side runtime.

2. Top-level architecture split

The repo is organized around two major phases:

- Host-side model preparation and compilation: converting, quantizing, pruning, and compiling models into runtime artifacts.

- Target-side inference execution: C++ sample apps and runtime wrappers that load the compiled artifacts and execute inference on board hardware.

3. Compiler and build toolchain

The setup/, tvm/, 3rdparty/, tutorials/, and obj/ areas form the build/compile layer. This is where TVM integration, MERA compiler support, runtime headers, and prebuilt runtime binaries live. The repository is set up to build a TVM-based compiler environment for the supported RZ/V targets.

4. Model compilation pipeline

The tutorials/ directory is the primary host-side entry point for model compilation. It contains scripts for ONNX, PyTorch, TFLite, and ExIR models, including quantized variants. These scripts drive the model import → optional preprocessing/quantization → DRP-AI compile flow and are the main place where model graphs are transformed into deployable runtime packages.

5. Preprocessing and operator adaptation

tutorials/drpai_preprocess/ contains the preprocessing logic that adapts models for DRP-AI constraints. tutorials/exclude_operator/ shows another architectural boundary: models may need operator exclusions or graph adjustments before compilation to fit the accelerator/runtime capabilities.

6. Conversion utilities and model acquisition

convert/ contains helper scripts for downloading reference models and converting between formats such as PyTorch, ONNX, and TFLite. This layer is upstream of compilation: it prepares model inputs in a compiler-friendly format and serves as a bridge from external model ecosystems into the RUHMI pipeline.

7. Runtime on the target board

apps/ is the board-side inference layer. It contains C++ runtime wrappers (MeraDrpRuntimeWrapper, PreRuntime, PreRuntimeV2H) plus sample executables and toolchain files. This layer consumes the compiled model artifacts and handles inference execution on the embedded device, including board-specific variants for V2H versus the other RZ/V families.

8. Application structure and boundary

The apps/ code separates generic runtime handling from model/sample-specific logic. The wrappers abstract model loading and execution, while sample programs demonstrate classification, benchmarking, and image-based inference. The apps/python/ directory provides a Python inference entry point and packaged wheels, showing that the runtime can also be accessed from Python on supported platforms.

9. How-to sample applications

how-to/sample_app/ and how-to/sample_app_v2h/ are richer end-user demos that combine camera input, image conversion, model-specific postprocessing, and a websocket-based UI. These samples are architecturally important because they show the full runtime path: camera capture → preprocessing → inference → postprocessing → UI/telemetry output.

10. Model-specific inference modules

Inside the how-to sample apps, the recognize/ and recognize_proc layers encapsulate model adapters for tasks like classification, object detection, pose estimation, semantic segmentation, and face-related models. This is a key plug-in boundary: the app framework stays generic while per-model code handles input/output shapes and postprocessing.

11. Pruning and model optimization

pruning/ contains separate workflows for structured pruning and retraining, with examples for YOLOX, YOLOv8x, ResNet50, TopFormer, and ViT. This subsystem sits before compilation and is used to reduce model cost while keeping compatibility with the DRP-AI deployment path.

12. Quantization-aware training

QAT/ provides training scripts for PyTorch and TensorFlow. This is another upstream optimization path that feeds into the same compile/deploy pipeline, letting models be prepared for lower precision execution before they reach the compiler.

13. Documentation as architecture map

docs/ is not just reference material; it documents the supported model matrix, board-specific build paths, runtime wrappers, preprocessing behavior, and known errors. how-to/ provides task-oriented examples and is effectively the integration layer for specific model families and deployment patterns.

14. External and embedded dependencies

The repo vendors major dependencies such as TVM, protobuf, ONNX, DLPack, DMLC-core, and support libraries. These are not incidental: they define the compiler/runtime boundary, model IR support, and the embedded execution stack used to move from ML framework models to on-device inference.

15. End-to-end data flow

The typical flow is: external model source → conversion or download helper → optional pruning/QAT/preprocessing → DRP-AI TVM compile → generated runtime binaries under obj/ or app-specific assets → deployment to the RZ/V board → C++/Python runtime app executes inference on camera or file input → task-specific postprocessing and UI/output.

16. Main architectural takeaway

This repo is a deployment platform, not a single application. Its core abstraction is the compiled model artifact plus a thin board runtime, with separate host-side tooling for preparation and board-side apps for execution, all organized around the DRP-AI TVM/MERA pipeline for Renesas RZ/V hardware.