# priyadarshini75/sam3-tensorrt architecture

> Architecture diagram of the GitHub repository priyadarshini75/sam3-tensorrt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/priyadarshini75/sam3-tensorrt
- Repository: https://github.com/priyadarshini75/sam3-tensorrt

## Overview

This repository has two major planes of activity: a **model preparation/build pipeline** in Python and a **runtime inference pipeline** in C++/CUDA/TensorRT. The architecture is centered on taking Meta’s SAM3 model, exporting it into a TensorRT-friendly form, and then running fast promptable segmentation through a native C++ library and demo app.

#### 1) High-level architecture

The repo is organized around these main boundaries:

- **External model source**: Hugging Face `facebook/sam3`
- **Python tooling**: exports and utilities
- **TensorRT build step**: converts ONNX into a serialized engine
- **C++/CUDA runtime**: performs inference and image postprocessing
- **Demo/application layer**: command-line app that drives segmentation on image folders
- **Containerized environment**: reproducible builds for x86 and aarch64/Jetson/Spark

A useful way to draw the architecture is as a pipeline:

1. User provides a Hugging Face token and runs the Python exporter.
2. Exporter pulls SAM3 weights from Hugging Face and writes ONNX plus external weight shards.
3. TensorRT builds a `.plan` engine from the ONNX.
4. The C++ demo app loads the engine, preprocesses images, runs inference, and postprocesses masks.
5. Results are saved to disk, and optional visualization scripts can render or inspect outputs.

#### 2) Python subsystem: model export and benchmarking

##### `python/onnxexport.py`
This is the **model conversion boundary**. It is responsible for taking SAM3 from the Python/model ecosystem and exporting a “clean” ONNX graph suitable for TensorRT.

Key architectural characteristics:
- It depends on access to the gated Hugging Face model.
- It uses the Hugging Face token from the environment.
- The export produces **external data shards**, not a single monolithic ONNX file, because SAM3 is large.
- The output directory is `onnx_weights/`, and the full directory must be kept together for downstream TensorRT parsing.

This script is part of the build pipeline, not the runtime path.

##### `python/basic_script.py`
This is the **baseline/reference execution path** for Hugging Face + PyTorch. It is used for benchmarking against the TensorRT/CUDA runtime.

Architecturally, this script serves as:
- a correctness and performance comparison point,
- a GPU-backed baseline,
- a way to measure the end-to-end segmentation pipeline without TensorRT.

##### `python/visualize.py`
This is an auxiliary visualization layer for inspecting segmentation outputs. It is not central to the inference backend, but it belongs to the “results interpretation” side of the repo.

#### 3) TensorRT build step: external to the library

The repo explicitly separates **engine building** from **engine execution**.

- The ONNX export is produced by Python.
- The TensorRT engine is built with `trtexec`, not by the C++ library.
- The library expects an already-built engine file such as `sam3_fp16.plan`.

This is an important architectural boundary:
- **Build-time optimization** happens outside the C++ code.
- **Runtime inference** is handled by the native library.

This separation keeps the C++ layer focused on execution, preprocessing, and postprocessing, rather than model compilation.

#### 4) C++/CUDA runtime subsystem

The `cpp/` directory contains the native inference library and its app.

##### Public interface: `cpp/include/`
These headers define the library boundary exposed to consumers.

- `sam3.hpp`: public C++ interface / API surface
- `sam3.cuh`: CUDA/TensorRT-facing declarations for the SAM3 runtime
- `prepost.cuh`: preprocessing and postprocessing kernel declarations

This suggests the library is split into:
- a user-facing API layer,
- a CUDA/TensorRT implementation layer,
- and a pre/postprocessing layer.

##### Implementation: `cpp/src/sam3/sam3_trt/`
This is the core runtime engine implementation.

- `sam3.cu`: main TensorRT inference logic for SAM3
- `prepost.cu`: CUDA preprocessing/postprocessing kernels

Architecturally, this layer does the heavy lifting:
- loading and using the TensorRT engine,
- preparing inputs,
- running inference on GPU,
- converting raw outputs into usable segmentation masks/logits.

The README indicates support for:
- **semantic segmentation**
- **instance segmentation**
- **Promptable Concept Segmentation (PCS)**

The postprocessing behavior matters:
- the library can emit mask logits,
- if visualization is disabled, the caller may need to apply sigmoid manually.

That implies the runtime is designed as a low-level backend with some responsibility intentionally left to the caller.

#### 5) Demo application subsystem

##### `cpp/src/sam3/sam3_apps/sam3_pcs_app.cpp`
This is the sample application and main consumer of the library.

Its role in the architecture:
- accepts an image directory and TensorRT engine path,
- invokes the `sam3_trt` library,
- runs semantic/instance segmentation workflows,
- writes outputs to a `results/` folder.

The demo app is effectively the application layer above the reusable runtime library.

The README also shows a benchmarking mode:
- when invoked with a benchmark flag, output writing is disabled,
- this isolates inference/performance measurement from disk I/O.

#### 6) Data flow through the runtime path

A typical inference flow looks like this:

1. **Input acquisition**
   - The app reads images from a directory.
   - A prompt drives segmentation behavior, such as text prompts like “dog” or “box”.

2. **Preprocessing**
   - CUDA preprocessing prepares the images for the SAM3/TensorRT input format.
   - On unified-memory platforms like Jetson or DGX Spark, the repo supports **zero-copy** paths, which reduce CPU/GPU transfer overhead.

3. **Inference**
   - The app passes inputs into the `sam3_trt` library.
   - The library executes the TensorRT engine.

4. **Postprocessing**
   - CUDA postprocessing transforms model outputs into semantic/instance segmentation outputs.
   - Depending on visualization mode, the output may be raw logits or finalized masks.

5. **Output**
   - Results are written to `results/`.
   - Optional visualization scripts or external tooling can inspect the masks.

#### 7) Platform and deployment boundary

##### `docker/`
There are separate Dockerfiles for:
- `Dockerfile.x86`
- `Dockerfile.aarch64`

This is a significant architectural piece because the repo is designed to run in a reproducible environment across:
- x86 GPUs
- Jetson / Spark / other aarch64 unified-memory systems

The container boundary encapsulates:
- CUDA,
- TensorRT,
- OpenCV,
- model export dependencies,
- runtime dependencies.

The README notes that on aarch64 unified-memory systems, the library can take advantage of zero-copy inference paths. That is a deployment-specific optimization that affects the data movement architecture between host and device memory.

#### 8) Build system boundary

##### `cpp/CMakeLists.txt`
CMake builds the native library and demo app.

The README indicates:
- shared library target: `sam3_trt`
- app target: `sam3_pcs_app`

So the native build produces at least:
- a reusable inference library,
- an executable demo/benchmark application.

#### 9) Important external dependencies and services

These are not incidental; they shape the architecture:

- **Hugging Face**: source of the gated SAM3 model
- **TensorRT**: engine compilation and runtime execution
- **CUDA**: custom preprocessing/postprocessing and GPU acceleration
- **OpenCV**: image handling in the C++ layer
- **Docker**: environment reproducibility
- **trtexec**: engine build and quick benchmarking tool

#### 10) Architectural constraints and design choices

Several design decisions are worth showing explicitly in a diagram:

- The repo is **not** a training repo; it is an export/build/inference repo.
- The TensorRT engine is an external artifact, not generated by the library itself.
- Large model export requires **ONNX external data shards**.
- The runtime is split between **library** and **demo app**, so the library can be embedded in other applications.
- Zero-copy support is a platform-specific optimization for unified-memory systems.
- Benchmarking is supported in both the PyTorch baseline and the TensorRT/CUDA path for apples-to-apples comparison.

#### 11) Suggested diagram layers

If you are drawing this repository, the cleanest layout is:

- **Top layer**: user/application inputs
  - image directory
  - text prompt
  - HF token
  - engine path

- **Build/prep layer**:
  - `onnxexport.py`
  - ONNX + external weights
  - `trtexec` engine build

- **Runtime layer**:
  - `sam3_pcs_app`
  - `sam3_trt` library
  - CUDA pre/post kernels
  - TensorRT engine

- **Support layer**:
  - Docker x86 / aarch64
  - OpenCV, CUDA, TensorRT, HF, CMake

That captures the repository’s real architecture: a model-export pipeline feeding a native TensorRT/CUDA inference backend with a thin demo application on top.

Main entry points: [`cpp/CMakeLists.txt`](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/cpp/CMakeLists.txt) (sam3_trt), [`cpp/src/sam3/sam3_apps/sam3_pcs_app.cpp`](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/cpp/src/sam3/sam3_apps/sam3_pcs_app.cpp) (PCS app), [`cpp/src/sam3/sam3_trt/sam3.cu`](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/cpp/src/sam3/sam3_trt/sam3.cu) (Inference core), [`cpp/src/sam3/sam3_trt/prepost.cu`](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/cpp/src/sam3/sam3_trt/prepost.cu) (Pre/post kernels)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph "External deps"
  hf_sam3(("SAM3 model<br/>model source"))
  trtexec["trtexec<br/>engine builder"]
  opencv["OpenCV<br/>image IO"]
  cuda_rt["CUDA/TensorRT<br/>gpu runtime"]
end

subgraph "Python prep"
  onnxexport["ONNX export<br/>python export<br/>[onnxexport.py]"]
  basic_script["Baseline run<br/>python benchmark<br/>[basic_script.py]"]
  visualize["Visualize<br/>python viz<br/>[visualize.py]"]
  onnx_weights[("ONNX shards<br/>export artifact")]
end

subgraph "Native runtime"
  sam3_hpp["SAM3 API<br/>public header<br/>[sam3.hpp]"]
  sam3_cuh["SAM3 CUDA API<br/>cuda header<br/>[sam3.cuh]"]
  prepost_cuh["Pre/post API<br/>kernel header<br/>[prepost.cuh]"]
  sam3_trt_cu["Inference core<br/>cuda runtime<br/>[sam3.cu]"]
  prepost_cu["Pre/post kernels<br/>cuda kernels<br/>[prepost.cu]"]
  sam3_lib["sam3_trt<br/>shared library<br/>[CMakeLists.txt]"]
  sam3_app["PCS app<br/>demo app<br/>[sam3_pcs_app.cpp]"]
end

subgraph "Deployment"
  docker_x86["x86 image<br/>container<br/>[Dockerfile.x86]"]
  docker_aarch64["aarch64 image<br/>container<br/>[Dockerfile.aarch64]"]
end

hf_sam3 -->|"model source"| onnxexport
onnxexport -->|"writes"| onnx_weights
onnx_weights -->|"input"| trtexec
trtexec -->|"builds engine for"| sam3_lib
basic_script -->|"baseline inference"| hf_sam3
sam3_app -->|"calls"| sam3_lib
sam3_lib -->|"implements"| sam3_trt_cu
sam3_lib -->|"uses"| prepost_cu
sam3_hpp -->|"public API"| sam3_lib
sam3_cuh -->|"declares"| sam3_trt_cu
prepost_cuh -->|"declares"| prepost_cu
sam3_app -->|"loads images"| opencv
sam3_app -->|"runs on"| cuda_rt
sam3_trt_cu -->|"executes on"| cuda_rt
prepost_cu -->|"accelerates"| cuda_rt
sam3_app -.->|"optional inspect"| visualize
docker_x86 -->|"packages"| sam3_lib
docker_aarch64 -->|"packages"| sam3_lib

click onnxexport "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/python/onnxexport.py"
click basic_script "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/python/basic_script.py"
click visualize "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/python/visualize.py"
click sam3_hpp "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/cpp/include/sam3.hpp"
click sam3_cuh "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/cpp/include/sam3.cuh"
click prepost_cuh "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/cpp/include/prepost.cuh"
click sam3_trt_cu "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/cpp/src/sam3/sam3_trt/sam3.cu"
click prepost_cu "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/cpp/src/sam3/sam3_trt/prepost.cu"
click sam3_lib "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/cpp/CMakeLists.txt"
click sam3_app "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/cpp/src/sam3/sam3_apps/sam3_pcs_app.cpp"
click docker_x86 "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/docker/Dockerfile.x86"
click docker_aarch64 "https://github.com/priyadarshini75/sam3-tensorrt/blob/main/docker/Dockerfile.aarch64"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class hf_sam3,trtexec,opencv,cuda_rt toneBlue
class onnxexport,basic_script,visualize,onnx_weights toneAmber
class sam3_hpp,sam3_cuh,prepost_cuh,sam3_trt_cu,prepost_cu,sam3_lib,sam3_app toneMint
class docker_x86,docker_aarch64 toneRose
```

## Components

### External deps

Model source and runtime/tooling outside the repo

- **SAM3 model**: model source · Gated Hugging Face model used as the source for export
- **trtexec**: engine builder · External TensorRT tool used to compile ONNX into a plan engine
- **OpenCV**: image IO · External image handling dependency for the native stack
- **CUDA/TensorRT**: gpu runtime · GPU runtime stack used by preprocessing, inference, and postprocessing

### Python prep

Model export and reference scripts

- [ONNX export](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/python/onnxexport.py): python export · Exports SAM3 to ONNX plus external weight shards · `python/onnxexport.py`
- [Baseline run](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/python/basic_script.py): python benchmark · PyTorch/Hugging Face reference path for correctness and performance comparison · `python/basic_script.py`
- [Visualize](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/python/visualize.py): python viz · Optional output inspection and rendering helper · `python/visualize.py`
- **ONNX shards**: export artifact · External-data ONNX package produced by export and consumed by engine build

### Native runtime

C++/CUDA inference library and demo app

- [SAM3 API](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/cpp/include/sam3.hpp): public header · Public C++ interface for consumers of the runtime library · `cpp/include/sam3.hpp`
- [SAM3 CUDA API](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/cpp/include/sam3.cuh): cuda header · CUDA/TensorRT-facing declarations for the runtime implementation · `cpp/include/sam3.cuh`
- [Pre/post API](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/cpp/include/prepost.cuh): kernel header · Preprocessing and postprocessing kernel declarations · `cpp/include/prepost.cuh`
- [Inference core](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/cpp/src/sam3/sam3_trt/sam3.cu): cuda runtime · TensorRT engine loading and GPU inference implementation · `cpp/src/sam3/sam3_trt/sam3.cu`
- [Pre/post kernels](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/cpp/src/sam3/sam3_trt/prepost.cu): cuda kernels · GPU preprocessing and postprocessing implementation · `cpp/src/sam3/sam3_trt/prepost.cu`
- [sam3_trt](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/cpp/CMakeLists.txt): shared library · Reusable native inference library built by CMake · `cpp/CMakeLists.txt`
- [PCS app](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/cpp/src/sam3/sam3_apps/sam3_pcs_app.cpp): demo app · Command-line segmentation application over image folders · `cpp/src/sam3/sam3_apps/sam3_pcs_app.cpp`

### Deployment

Container and build system boundaries

- [x86 image](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/docker/Dockerfile.x86): container · Container build for x86 CUDA/TensorRT environments · `docker/Dockerfile.x86`
- [aarch64 image](https://github.com/priyadarshini75/sam3-tensorrt/blob/HEAD/docker/Dockerfile.aarch64): container · Container build for Jetson/Spark and other aarch64 targets · `docker/Dockerfile.aarch64`

## Connections

- SAM3 model → ONNX export: model source · Loads gated SAM3 weights for export
- ONNX export → ONNX shards: writes · Produces ONNX plus external weight shards
- ONNX shards → trtexec: input · Feeding the exported model into TensorRT engine build
- trtexec → sam3_trt: builds engine for · Creates the serialized plan consumed at runtime
- Baseline run → SAM3 model: baseline inference · Reference PyTorch path against the same model source
- PCS app → sam3_trt: calls · Application uses the reusable native inference library
- sam3_trt → Inference core: implements · Core TensorRT/CUDA execution path
- sam3_trt → Pre/post kernels: uses · Pre/postprocessing kernels are part of the runtime library
- SAM3 API → sam3_trt: public API · Header boundary for consumers of the shared library
- SAM3 CUDA API → Inference core: declares · CUDA-facing declarations for the inference core
- Pre/post API → Pre/post kernels: declares · Kernel interface for preprocessing and postprocessing
- PCS app → OpenCV: loads images · Uses OpenCV-backed image handling in the demo pipeline
- PCS app → CUDA/TensorRT: runs on · Depends on CUDA/TensorRT for accelerated inference
- Inference core → CUDA/TensorRT: executes on · Inference core runs on the GPU runtime stack
- Pre/post kernels → CUDA/TensorRT: accelerates · Pre/postprocessing kernels run on CUDA
- PCS app → Visualize: optional inspect · Outputs can be inspected with auxiliary visualization
- x86 image → sam3_trt: packages · Containerized build/runtime environment for native stack
- aarch64 image → sam3_trt: packages · Containerized build/runtime environment for unified-memory targets
