# aimol-l/sam2export architecture

> Architecture diagram of the GitHub repository aimol-l/sam2export, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aimol-l/sam2export
- Repository: https://github.com/aimol-l/sam2export

## Overview

1. **Repository purpose**
   - This repo is a **SAM2-to-ONNX export bundle**. It vendors the SAM2 model implementation and adds an export script so the model can be converted from PyTorch checkpoints into ONNX artifacts for deployment.

2. **Primary entry point**
   - `export_onnx.py` is the top-level executable. It likely loads a SAM2 config, restores a checkpoint, builds the model, and traces/saves ONNX outputs.
   - The README shows the intended flow: download checkpoints, create model-size folders, then run the export script.

3. **Model construction boundary**
   - `sam2/build_sam.py` and `sam2/modeling/sam2_base.py` are the main assembly layer.
   - This layer wires together the encoder, prompt handling, mask decoding, and video-memory components into a full SAM2 model instance.
   - `sam2_configs/*.yaml` defines the model variants: `tiny`, `small`, `base+`, and `large`.

4. **Image segmentation path**
   - `sam2/sam2_image_predictor.py` is the image-facing inference wrapper.
   - It routes an image through preprocessing, image encoding, prompt encoding, and mask decoding.
   - `sam2/automatic_mask_generator.py` provides the “no prompt” generation path that repeatedly queries the model to produce masks over an image.

5. **Video segmentation path**
   - `sam2/sam2_video_predictor.py` is the temporal inference wrapper.
   - This path adds state across frames, using memory features to propagate object identity and segmentation quality over time.
   - The architecture boundary here is important: video inference is not just repeated image inference; it depends on memory encoding and memory attention.

6. **Core neural subsystems**
   - `sam2/modeling/backbones/` contains the **image backbone**, centered on `hieradet.py` and `image_encoder.py`.
   - `sam2/modeling/sam/` contains the **prompt encoder**, **mask decoder**, and transformer blocks used to turn prompts + image features into segmentation masks.
   - `sam2/modeling/position_encoding.py` and `sam2/modeling/memory_attention.py` support spatial and temporal reasoning.

7. **Temporal memory subsystem**
   - `sam2/modeling/memory_encoder.py` and `sam2/modeling/memory_attention.py` are the key video-specific pieces.
   - They encode past predictions into a memory representation and attend to that memory when processing later frames.
   - This is the main architectural difference between SAM2 and a pure image segmentation model.

8. **Custom / low-level performance code**
   - `sam2/csrc/connected_components.cu` is a CUDA extension for connected-component operations.
   - This is likely used in post-processing or mask cleanup and represents the only obvious native acceleration layer in the repo.

9. **Utility and preprocessing layer**
   - `sam2/utils/` holds reusable helpers such as image transforms, miscellaneous utilities, and automatic mask generation helpers.
   - These utilities sit at the edge of the model, handling input normalization, resizing, and output formatting rather than core inference logic.

10. **Checkpoint and export assets**
   - `checkpoints/download_ckpts.sh` is the model asset acquisition boundary; it fetches pretrained weights from external sources.
   - The downloaded checkpoints are then consumed by the export pipeline.
   - The repo is therefore split between **static model code** and **external weight files**.

11. **Repository packaging**
   - `sam2/__init__.py` exposes the vendored SAM2 implementation as a Python package.
   - `src/Module.py` appears to be a small local wrapper or helper module outside the main SAM2 package, likely used to adapt the model for export or integration.

12. **Architecture diagram takeaway**
   - A good diagram for this repo should show:
     - **Export script** at the top.
     - **Checkpoint/config inputs** feeding model construction.
     - A shared **SAM2 core model** split into **image encoder**, **prompt/mask decoder**, and **video memory path**.
     - Separate **image prediction**, **automatic mask generation**, and **video prediction** usage paths.
     - A small **CUDA post-processing** sidecar and a **utility/preprocessing** layer around the edges.

Main entry points: [`sam2/modeling/sam2_base.py`](https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/sam2_base.py) (SAM2 base), [`sam2/sam2_video_predictor.py`](https://github.com/aimol-l/sam2export/blob/main/sam2/sam2_video_predictor.py) (Video predictor), [`sam2/sam2_image_predictor.py`](https://github.com/aimol-l/sam2export/blob/main/sam2/sam2_image_predictor.py) (Image predictor), [`sam2/build_sam.py`](https://github.com/aimol-l/sam2export/blob/main/sam2/build_sam.py) (Build SAM2)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_entry["Export entrypoint"]
  node_n_export["Export ONNX<br/>entry script<br/>[export_onnx.py]"]
  node_n_ckpts["Checkpoints<br/>asset fetch<br/>[download_ckpts.sh]"]
  node_n_cfgs["Model configs<br/>yaml variants"]
end

subgraph group_g_model["SAM2 core"]
  node_n_build["Build SAM2<br/>assembly layer<br/>[build_sam.py]"]
  node_n_base["SAM2 base<br/>model shell<br/>[sam2_base.py]"]
  node_n_backbone["Image encoder<br/>vision backbone<br/>[image_encoder.py]"]
  node_n_hieradet["Hiera backbone<br/>hierarchical trunk<br/>[hieradet.py]"]
  node_n_prompt["Prompt encoder<br/>prompt module<br/>[prompt_encoder.py]"]
  node_n_decoder["Mask decoder<br/>segmentation head<br/>[mask_decoder.py]"]
  node_n_xform["Transformer blocks<br/>attention core<br/>[transformer.py]"]
  node_n_posenc["Position encoding<br/>spatial encoding"]
end

subgraph group_g_image["Image inference"]
  node_n_imgpred["Image predictor<br/>image inference"]
  node_n_amg["Auto mask gen<br/>prompt-free inference"]
end

subgraph group_g_video["Video inference"]
  node_n_video["Video predictor<br/>temporal inference"]
  node_n_memenc["Memory encoder<br/>temporal memory<br/>[memory_encoder.py]"]
  node_n_memattn["Memory attention<br/>temporal attention"]
end

subgraph group_g_support["Support code"]
  node_n_utils["Utils<br/>preprocess helpers<br/>[transforms.py]"]
  node_n_csrc{{"Connected comps<br/>cuda extension"}}
  node_n_pkg["SAM2 package<br/>python package<br/>[__init__.py]"]
  node_n_module["Local module<br/>adapter wrapper<br/>[Module.py]"]
end

node_n_ckpts -->|"weights"| node_n_export
node_n_cfgs -->|"selects"| node_n_build
node_n_export -->|"invokes"| node_n_build
node_n_build -->|"creates"| node_n_base
node_n_base -->|"uses"| node_n_backbone
node_n_base -->|"uses"| node_n_prompt
node_n_base -->|"uses"| node_n_decoder
node_n_base -->|"uses"| node_n_memenc
node_n_base -->|"uses"| node_n_memattn
node_n_backbone -->|"built on"| node_n_hieradet
node_n_backbone -->|"adds"| node_n_posenc
node_n_prompt -->|"runs through"| node_n_xform
node_n_decoder -->|"uses"| node_n_xform
node_n_imgpred -->|"calls"| node_n_base
node_n_imgpred -->|"preprocesses"| node_n_utils
node_n_amg -->|"builds on"| node_n_imgpred
node_n_amg -->|"postprocesses"| node_n_csrc
node_n_video -->|"calls"| node_n_base
node_n_video -->|"stores"| node_n_memenc
node_n_video -->|"attends"| node_n_memattn
node_n_module -->|"adapts"| node_n_export
node_n_pkg -->|"exposes"| node_n_build
node_n_pkg -->|"exposes"| node_n_imgpred
node_n_pkg -->|"exposes"| node_n_video

click node_n_export "https://github.com/aimol-l/sam2export/blob/main/export_onnx.py"
click node_n_ckpts "https://github.com/aimol-l/sam2export/blob/main/checkpoints/download_ckpts.sh"
click node_n_cfgs "https://github.com/aimol-l/sam2export/blob/main/sam2_configs/sam2_hiera_tiny.yaml"
click node_n_build "https://github.com/aimol-l/sam2export/blob/main/sam2/build_sam.py"
click node_n_base "https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/sam2_base.py"
click node_n_backbone "https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/backbones/image_encoder.py"
click node_n_hieradet "https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/backbones/hieradet.py"
click node_n_prompt "https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/sam/prompt_encoder.py"
click node_n_decoder "https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/sam/mask_decoder.py"
click node_n_xform "https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/sam/transformer.py"
click node_n_posenc "https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/position_encoding.py"
click node_n_imgpred "https://github.com/aimol-l/sam2export/blob/main/sam2/sam2_image_predictor.py"
click node_n_amg "https://github.com/aimol-l/sam2export/blob/main/sam2/automatic_mask_generator.py"
click node_n_video "https://github.com/aimol-l/sam2export/blob/main/sam2/sam2_video_predictor.py"
click node_n_memenc "https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/memory_encoder.py"
click node_n_memattn "https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/memory_attention.py"
click node_n_utils "https://github.com/aimol-l/sam2export/blob/main/sam2/utils/transforms.py"
click node_n_csrc "https://github.com/aimol-l/sam2export/blob/main/sam2/csrc/connected_components.cu"
click node_n_pkg "https://github.com/aimol-l/sam2export/blob/main/sam2/__init__.py"
click node_n_module "https://github.com/aimol-l/sam2export/blob/main/src/Module.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_export,node_n_ckpts,node_n_cfgs toneBlue
class node_n_build,node_n_base,node_n_backbone,node_n_hieradet,node_n_prompt,node_n_decoder,node_n_xform,node_n_posenc toneAmber
class node_n_imgpred,node_n_amg toneMint
class node_n_video,node_n_memenc,node_n_memattn toneRose
class node_n_utils,node_n_csrc,node_n_pkg,node_n_module toneIndigo
```

## Components

### Export entrypoint

Top-level script and external assets that drive conversion to ONNX.

- [Export ONNX](https://github.com/aimol-l/sam2export/blob/main/export_onnx.py): entry script · Top-level executable that loads configs and checkpoints, builds the model, and exports ONNX artifacts. · `export_onnx.py`
- [Checkpoints](https://github.com/aimol-l/sam2export/blob/main/checkpoints/download_ckpts.sh): asset fetch · External pretrained weights acquired by the download helper and consumed by export. · `checkpoints/download_ckpts.sh`
- [Model configs](https://github.com/aimol-l/sam2export/blob/main/sam2_configs/sam2_hiera_tiny.yaml): yaml variants · SAM2 size and variant definitions for tiny, small, base+, and large models. · `sam2_configs/sam2_hiera_tiny.yaml`

### SAM2 core

Model assembly and the main neural subsystems used by both image and video inference.

- [Build SAM2](https://github.com/aimol-l/sam2export/blob/main/sam2/build_sam.py): assembly layer · Constructs a configured SAM2 model from the selected config and weights. · `sam2/build_sam.py`
- [SAM2 base](https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/sam2_base.py): model shell · Core model container that wires image, prompt, memory, and decoding subsystems together. · `sam2/modeling/sam2_base.py`
- [Image encoder](https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/backbones/image_encoder.py): vision backbone · Hierarchical image feature extractor used for segmentation inputs. · `sam2/modeling/backbones/image_encoder.py`
- [Hiera backbone](https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/backbones/hieradet.py): hierarchical trunk · Backbone implementation and supporting hierarchy used by the encoder. · `sam2/modeling/backbones/hieradet.py`
- [Prompt encoder](https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/sam/prompt_encoder.py): prompt module · Encodes points, boxes, and other prompts into model-ready embeddings. · `sam2/modeling/sam/prompt_encoder.py`
- [Mask decoder](https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/sam/mask_decoder.py): segmentation head · Transforms image and prompt features into mask predictions. · `sam2/modeling/sam/mask_decoder.py`
- [Transformer blocks](https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/sam/transformer.py): attention core · Transformer layers used inside the prompt-to-mask decoding path. · `sam2/modeling/sam/transformer.py`
- [Position encoding](https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/position_encoding.py): spatial encoding · Adds spatial structure for image and prompt feature reasoning. · `sam2/modeling/position_encoding.py`

### Image inference

Single-frame inference and prompt-free mask generation paths.

- [Image predictor](https://github.com/aimol-l/sam2export/blob/main/sam2/sam2_image_predictor.py): image inference · Single-image inference wrapper that runs preprocessing, encoding, prompting, and decoding. · `sam2/sam2_image_predictor.py`
- [Auto mask gen](https://github.com/aimol-l/sam2export/blob/main/sam2/automatic_mask_generator.py): prompt-free inference · Generates masks by repeatedly querying the model over an image without manual prompts. · `sam2/automatic_mask_generator.py`

### Video inference

Temporal inference path built around memory and frame-to-frame propagation.

- [Video predictor](https://github.com/aimol-l/sam2export/blob/main/sam2/sam2_video_predictor.py): temporal inference · Frame-to-frame inference wrapper that maintains state and propagates object identity over time. · `sam2/sam2_video_predictor.py`
- [Memory encoder](https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/memory_encoder.py): temporal memory · Encodes past predictions into a representation that can be reused on later frames. · `sam2/modeling/memory_encoder.py`
- [Memory attention](https://github.com/aimol-l/sam2export/blob/main/sam2/modeling/memory_attention.py): temporal attention · Attends to stored memory features when processing new frames. · `sam2/modeling/memory_attention.py`

### Support code

Utilities, package surface, native acceleration, and local wrappers.

- [Utils](https://github.com/aimol-l/sam2export/blob/main/sam2/utils/transforms.py): preprocess helpers · Reusable transforms, miscellaneous helpers, and mask-generation utilities around the model edge. · `sam2/utils/transforms.py`
- [Connected comps](https://github.com/aimol-l/sam2export/blob/main/sam2/csrc/connected_components.cu): cuda extension · Native CUDA acceleration for connected-component style post-processing. · `sam2/csrc/connected_components.cu`
- [SAM2 package](https://github.com/aimol-l/sam2export/blob/main/sam2/__init__.py): python package · Vendored package entry surface for the SAM2 implementation. · `sam2/__init__.py`
- [Local module](https://github.com/aimol-l/sam2export/blob/main/src/Module.py): adapter wrapper · Small external wrapper used to adapt or integrate the exported model. · `src/Module.py`

## Connections

- Checkpoints → Export ONNX: weights · Export uses downloaded checkpoint files.
- Model configs → Build SAM2: selects · Model YAML selects the variant to assemble.
- Export ONNX → Build SAM2: invokes · Export entrypoint calls model construction.
- Build SAM2 → SAM2 base: creates · Builder instantiates the core SAM2 model.
- SAM2 base → Image encoder: uses · Core model wires in image feature extraction.
- SAM2 base → Prompt encoder: uses · Core model wires in prompt encoding.
- SAM2 base → Mask decoder: uses · Core model wires in mask decoding.
- SAM2 base → Memory encoder: uses · Video-capable model includes memory encoding.
- SAM2 base → Memory attention: uses · Video-capable model includes memory attention.
- Image encoder → Hiera backbone: built on · The encoder is implemented with the Hiera backbone.
- Image encoder → Position encoding: adds · Spatial encoding supports image features.
- Prompt encoder → Transformer blocks: runs through · Prompt features are processed by transformer blocks.
- Mask decoder → Transformer blocks: uses · Decoder relies on transformer internals.
- Image predictor → SAM2 base: calls · Image predictor runs the assembled SAM2 model.
- Image predictor → Utils: preprocesses · Image path depends on transforms and helpers.
- Auto mask gen → Image predictor: builds on · Automatic mask generation uses the image predictor path.
- Auto mask gen → Connected comps: postprocesses · Mask generation can rely on native connected-component acceleration.
- Video predictor → SAM2 base: calls · Video predictor runs the same core model with temporal state.
- Video predictor → Memory encoder: stores · Video path encodes past outputs into memory.
- Video predictor → Memory attention: attends · Video path retrieves memory when processing later frames.
- Local module → Export ONNX: adapts · Local wrapper likely bridges the model into export or integration flow.
- SAM2 package → Build SAM2: exposes · Package surface makes the implementation importable for assembly.
- SAM2 package → Image predictor: exposes · Package surface exposes image inference APIs.
- SAM2 package → Video predictor: exposes · Package surface exposes video inference APIs.
