How priyadarshini75/sam3-tensorrt works
This repository has two major planes of activity: a model preparation/build pipeline in Python and a runtime inference pipeline in C++/CUDA/TensorRT. The architecture is centered on taking Meta’s SAM3 model, exporting it into a TensorRT-friendly form, and then running fast promptable segmentation through a native C++ library and demo app.
1) High-level architecture
The repo is organized around these main boundaries:
• External model source: Hugging Face facebook/sam3
• Python tooling: exports and utilities
• TensorRT build step: converts ONNX into a serialized engine
• C++/CUDA runtime: performs inference and image postprocessing
• Demo/application layer: command-line app that drives segmentation on image folders
• Containerized environment: reproducible builds for x86 and aarch64/Jetson/Spark
A useful way to draw the architecture is as a pipeline:
1. User provides a Hugging Face token and runs the Python exporter.
2. Exporter pulls SAM3 weights from Hugging Face and writes ONNX plus external weight shards.
3. TensorRT builds a .plan engine from the ONNX.
4. The C++ demo app loads the engine, preprocesses images, runs inference, and postprocesses masks.
5. Results are saved to disk, and optional visualization scripts can render or inspect outputs.
2) Python subsystem: model export and benchmarking
python/onnxexport.py
This is the model conversion boundary. It is responsible for taking SAM3 from the Python/model ecosystem and exporting a “clean” ONNX graph suitable for TensorRT.
Key architectural characteristics:
• It depends on access to the gated Hugging Face model.
• It uses the Hugging Face token from the environment.
• The export produces external data shards, not a single monolithic ONNX file, because SAM3 is large.
• The output directory is onnx_weights/, and the full directory must be kept together for downstream TensorRT parsing.
This script is part of the build pipeline, not the runtime path.
python/basic_script.py
This is the baseline/reference execution path for Hugging Face + PyTorch. It is used for benchmarking against the TensorRT/CUDA runtime.
Architecturally, this script serves as:
• a correctness and performance comparison point,
• a GPU-backed baseline,
• a way to measure the end-to-end segmentation pipeline without TensorRT.
python/visualize.py
This is an auxiliary visualization layer for inspecting segmentation outputs. It is not central to the inference backend, but it belongs to the “results interpretation” side of the repo.
3) TensorRT build step: external to the library
The repo explicitly separates engine building from engine execution.
• The ONNX export is produced by Python.
• The TensorRT engine is built with trtexec, not by the C++ library.
• The library expects an already-built engine file such as sam3_fp16.plan.
This is an important architectural boundary:
• Build-time optimization happens outside the C++ code.
• Runtime inference is handled by the native library.
This separation keeps the C++ layer focused on execution, preprocessing, and postprocessing, rather than model compilation.
4) C++/CUDA runtime subsystem
The cpp/ directory contains the native inference library and its app.
Public interface: cpp/include/
These headers define the library boundary exposed to consumers.
• sam3.hpp: public C++ interface / API surface
• sam3.cuh: CUDA/TensorRT-facing declarations for the SAM3 runtime
• prepost.cuh: preprocessing and postprocessing kernel declarations
This suggests the library is split into:
• a user-facing API layer,
• a CUDA/TensorRT implementation layer,
• and a pre/postprocessing layer.
Implementation: cpp/src/sam3/sam3_trt/
This is the core runtime engine implementation.
• sam3.cu: main TensorRT inference logic for SAM3
• prepost.cu: CUDA preprocessing/postprocessing kernels
Architecturally, this layer does the heavy lifting:
• loading and using the TensorRT engine,
• preparing inputs,
• running inference on GPU,
• converting raw outputs into usable segmentation masks/logits.
The README indicates support for:
• semantic segmentation
• instance segmentation
• Promptable Concept Segmentation (PCS)
The postprocessing behavior matters:
• the library can emit mask logits,
• if visualization is disabled, the caller may need to apply sigmoid manually.
That implies the runtime is designed as a low-level backend with some responsibility intentionally left to the caller.
5) Demo application subsystem
cpp/src/sam3/sam3_apps/sam3_pcs_app.cpp
This is the sample application and main consumer of the library.
Its role in the architecture:
• accepts an image directory and TensorRT engine path,
• invokes the sam3_trt library,
• runs semantic/instance segmentation workflows,
• writes outputs to a results/ folder.
The demo app is effectively the application layer above the reusable runtime library.
The README also shows a benchmarking mode:
• when invoked with a benchmark flag, output writing is disabled,
• this isolates inference/performance measurement from disk I/O.
6) Data flow through the runtime path
A typical inference flow looks like this:
1. Input acquisition
- The app reads images from a directory.
- A prompt drives segmentation behavior, such as text prompts like “dog” or “box”.
2. Preprocessing
- CUDA preprocessing prepares the images for the SAM3/TensorRT input format.
- On unified-memory platforms like Jetson or DGX Spark, the repo supports zero-copy paths, which reduce CPU/GPU transfer overhead.
3. Inference
- The app passes inputs into the sam3_trt library.
- The library executes the TensorRT engine.
4. Postprocessing
- CUDA postprocessing transforms model outputs into semantic/instance segmentation outputs.
- Depending on visualization mode, the output may be raw logits or finalized masks.
5. Output
- Results are written to results/.
- Optional visualization scripts or external tooling can inspect the masks.
7) Platform and deployment boundary
docker/
There are separate Dockerfiles for:
• Dockerfile.x86
• Dockerfile.aarch64
This is a significant architectural piece because the repo is designed to run in a reproducible environment across:
• x86 GPUs
• Jetson / Spark / other aarch64 unified-memory systems
The container boundary encapsulates:
• CUDA,
• TensorRT,
• OpenCV,
• model export dependencies,
• runtime dependencies.
The README notes that on aarch64 unified-memory systems, the library can take advantage of zero-copy inference paths. That is a deployment-specific optimization that affects the data movement architecture between host and device memory.
8) Build system boundary
cpp/CMakeLists.txt
CMake builds the native library and demo app.
The README indicates:
• shared library target: sam3_trt
• app target: sam3_pcs_app
So the native build produces at least:
• a reusable inference library,
• an executable demo/benchmark application.
9) Important external dependencies and services
These are not incidental; they shape the architecture:
• Hugging Face: source of the gated SAM3 model
• TensorRT: engine compilation and runtime execution
• CUDA: custom preprocessing/postprocessing and GPU acceleration
• OpenCV: image handling in the C++ layer
• Docker: environment reproducibility
• trtexec: engine build and quick benchmarking tool
10) Architectural constraints and design choices
Several design decisions are worth showing explicitly in a diagram:
• The repo is not a training repo; it is an export/build/inference repo.
• The TensorRT engine is an external artifact, not generated by the library itself.
• Large model export requires ONNX external data shards.
• The runtime is split between library and demo app, so the library can be embedded in other applications.
• Zero-copy support is a platform-specific optimization for unified-memory systems.
• Benchmarking is supported in both the PyTorch baseline and the TensorRT/CUDA path for apples-to-apples comparison.
11) Suggested diagram layers
If you are drawing this repository, the cleanest layout is:
• Top layer: user/application inputs
- image directory
- text prompt
- HF token
- engine path
• Build/prep layer:
- onnxexport.py
- ONNX + external weights
- trtexec engine build
• Runtime layer:
- sam3_pcs_app
- sam3_trt library
- CUDA pre/post kernels
- TensorRT engine
• Support layer:
- Docker x86 / aarch64
- OpenCV, CUDA, TensorRT, HF, CMake
That captures the repository’s real architecture: a model-export pipeline feeding a native TensorRT/CUDA inference backend with a thin demo application on top.
Main entry points: CMakeLists.txt sam3_trt · sam3_pcs_app.cpp PCS app · sam3.cu Inference core · prepost.cu Pre/post kernels
Components 17
External deps
Model source and runtime/tooling outside the repo
- SAM3 model · Gated Hugging Face model used as the source for export
- trtexec · External TensorRT tool used to compile ONNX into a plan engine
- OpenCV · External image handling dependency for the native stack
- CUDA/TensorRT · GPU runtime stack used by preprocessing, inference, and postprocessing
Python prep
Model export and reference scripts
- ONNX export · Exports SAM3 to ONNX plus external weight shards
python/onnxexport.py - Baseline run · PyTorch/Hugging Face reference path for correctness and performance comparison
python/basic_script.py - Visualize · Optional output inspection and rendering helper
python/visualize.py - ONNX shards · External-data ONNX package produced by export and consumed by engine build
Native runtime
C++/CUDA inference library and demo app
- SAM3 API · Public C++ interface for consumers of the runtime library
cpp/include/sam3.hpp - SAM3 CUDA API · CUDA/TensorRT-facing declarations for the runtime implementation
cpp/include/sam3.cuh - Pre/post API · Preprocessing and postprocessing kernel declarations
cpp/include/prepost.cuh - Inference core · TensorRT engine loading and GPU inference implementation
cpp/src/sam3/sam3_trt/sam3.cu - Pre/post kernels · GPU preprocessing and postprocessing implementation
cpp/src/sam3/sam3_trt/prepost.cu - sam3_trt · Reusable native inference library built by CMake
cpp/CMakeLists.txt - PCS app · Command-line segmentation application over image folders
cpp/src/sam3/sam3_apps/sam3_pcs_app.cpp
Deployment
Container and build system boundaries
- x86 image · Container build for x86 CUDA/TensorRT environments
docker/Dockerfile.x86 - aarch64 image · Container build for Jetson/Spark and other aarch64 targets
docker/Dockerfile.aarch64
Connections 18
- SAM3 model → ONNX export: model source · Loads gated SAM3 weights for export
- ONNX export → ONNX shards: writes · Produces ONNX plus external weight shards
- ONNX shards → trtexec: input · Feeding the exported model into TensorRT engine build
- trtexec → sam3_trt: builds engine for · Creates the serialized plan consumed at runtime
- Baseline run → SAM3 model: baseline inference · Reference PyTorch path against the same model source
- PCS app → sam3_trt: calls · Application uses the reusable native inference library
- sam3_trt → Inference core: implements · Core TensorRT/CUDA execution path
- sam3_trt → Pre/post kernels: uses · Pre/postprocessing kernels are part of the runtime library
- SAM3 API → sam3_trt: public API · Header boundary for consumers of the shared library
- SAM3 CUDA API → Inference core: declares · CUDA-facing declarations for the inference core
- Pre/post API → Pre/post kernels: declares · Kernel interface for preprocessing and postprocessing
- PCS app → OpenCV: loads images · Uses OpenCV-backed image handling in the demo pipeline
- PCS app → CUDA/TensorRT: runs on · Depends on CUDA/TensorRT for accelerated inference
- Inference core → CUDA/TensorRT: executes on · Inference core runs on the GPU runtime stack
- Pre/post kernels → CUDA/TensorRT: accelerates · Pre/postprocessing kernels run on CUDA
- PCS app → Visualize: optional inspect · Outputs can be inspected with auxiliary visualization
- x86 image → sam3_trt: packages · Containerized build/runtime environment for native stack
- aarch64 image → sam3_trt: packages · Containerized build/runtime environment for unified-memory targets