How aimol-l/sam2export works
1. Repository purpose
- This repo is a SAM2-to-ONNX export bundle. It vendors the SAM2 model implementation and adds an export script so the model can be converted from PyTorch checkpoints into ONNX artifacts for deployment.
2. Primary entry point
- export_onnx.py is the top-level executable. It likely loads a SAM2 config, restores a checkpoint, builds the model, and traces/saves ONNX outputs.
- The README shows the intended flow: download checkpoints, create model-size folders, then run the export script.
3. Model construction boundary
- sam2/build_sam.py and sam2/modeling/sam2_base.py are the main assembly layer.
- This layer wires together the encoder, prompt handling, mask decoding, and video-memory components into a full SAM2 model instance.
- sam2_configs/*.yaml defines the model variants: tiny, small, base+, and large.
4. Image segmentation path
- sam2/sam2_image_predictor.py is the image-facing inference wrapper.
- It routes an image through preprocessing, image encoding, prompt encoding, and mask decoding.
- sam2/automatic_mask_generator.py provides the “no prompt” generation path that repeatedly queries the model to produce masks over an image.
5. Video segmentation path
- sam2/sam2_video_predictor.py is the temporal inference wrapper.
- This path adds state across frames, using memory features to propagate object identity and segmentation quality over time.
- The architecture boundary here is important: video inference is not just repeated image inference; it depends on memory encoding and memory attention.
6. Core neural subsystems
- sam2/modeling/backbones/ contains the image backbone, centered on hieradet.py and image_encoder.py.
- sam2/modeling/sam/ contains the prompt encoder, mask decoder, and transformer blocks used to turn prompts + image features into segmentation masks.
- sam2/modeling/position_encoding.py and sam2/modeling/memory_attention.py support spatial and temporal reasoning.
7. Temporal memory subsystem
- sam2/modeling/memory_encoder.py and sam2/modeling/memory_attention.py are the key video-specific pieces.
- They encode past predictions into a memory representation and attend to that memory when processing later frames.
- This is the main architectural difference between SAM2 and a pure image segmentation model.
8. Custom / low-level performance code
- sam2/csrc/connected_components.cu is a CUDA extension for connected-component operations.
- This is likely used in post-processing or mask cleanup and represents the only obvious native acceleration layer in the repo.
9. Utility and preprocessing layer
- sam2/utils/ holds reusable helpers such as image transforms, miscellaneous utilities, and automatic mask generation helpers.
- These utilities sit at the edge of the model, handling input normalization, resizing, and output formatting rather than core inference logic.
10. Checkpoint and export assets
- checkpoints/download_ckpts.sh is the model asset acquisition boundary; it fetches pretrained weights from external sources.
- The downloaded checkpoints are then consumed by the export pipeline.
- The repo is therefore split between static model code and external weight files.
11. Repository packaging
- sam2/__init__.py exposes the vendored SAM2 implementation as a Python package.
- src/Module.py appears to be a small local wrapper or helper module outside the main SAM2 package, likely used to adapt the model for export or integration.
12. Architecture diagram takeaway
- A good diagram for this repo should show:
- Export script at the top.
- Checkpoint/config inputs feeding model construction.
- A shared SAM2 core model split into image encoder, prompt/mask decoder, and video memory path.
- Separate image prediction, automatic mask generation, and video prediction usage paths.
- A small CUDA post-processing sidecar and a utility/preprocessing layer around the edges.
Main entry points: sam2_base.py SAM2 base · sam2_video_predictor.py Video predictor · sam2_image_predictor.py Image predictor · build_sam.py Build SAM2
Components 20
Export entrypoint
Top-level script and external assets that drive conversion to ONNX.
- Export ONNX · Top-level executable that loads configs and checkpoints, builds the model, and exports ONNX artifacts.
export_onnx.py - Checkpoints · External pretrained weights acquired by the download helper and consumed by export.
checkpoints/download_ckpts.sh - Model configs · SAM2 size and variant definitions for tiny, small, base+, and large models.
sam2_configs/sam2_hiera_tiny.yaml
SAM2 core
Model assembly and the main neural subsystems used by both image and video inference.
- Build SAM2 · Constructs a configured SAM2 model from the selected config and weights.
sam2/build_sam.py - SAM2 base · Core model container that wires image, prompt, memory, and decoding subsystems together.
sam2/modeling/sam2_base.py - Image encoder · Hierarchical image feature extractor used for segmentation inputs.
sam2/modeling/backbones/image_encoder.py - Hiera backbone · Backbone implementation and supporting hierarchy used by the encoder.
sam2/modeling/backbones/hieradet.py - Prompt encoder · Encodes points, boxes, and other prompts into model-ready embeddings.
sam2/modeling/sam/prompt_encoder.py - Mask decoder · Transforms image and prompt features into mask predictions.
sam2/modeling/sam/mask_decoder.py - Transformer blocks · Transformer layers used inside the prompt-to-mask decoding path.
sam2/modeling/sam/transformer.py - Position encoding · Adds spatial structure for image and prompt feature reasoning.
sam2/modeling/position_encoding.py
Image inference
Single-frame inference and prompt-free mask generation paths.
- Image predictor · Single-image inference wrapper that runs preprocessing, encoding, prompting, and decoding.
sam2/sam2_image_predictor.py - Auto mask gen · Generates masks by repeatedly querying the model over an image without manual prompts.
sam2/automatic_mask_generator.py
Video inference
Temporal inference path built around memory and frame-to-frame propagation.
- Video predictor · Frame-to-frame inference wrapper that maintains state and propagates object identity over time.
sam2/sam2_video_predictor.py - Memory encoder · Encodes past predictions into a representation that can be reused on later frames.
sam2/modeling/memory_encoder.py - Memory attention · Attends to stored memory features when processing new frames.
sam2/modeling/memory_attention.py
Support code
Utilities, package surface, native acceleration, and local wrappers.
- Utils · Reusable transforms, miscellaneous helpers, and mask-generation utilities around the model edge.
sam2/utils/transforms.py - Connected comps · Native CUDA acceleration for connected-component style post-processing.
sam2/csrc/connected_components.cu - SAM2 package · Vendored package entry surface for the SAM2 implementation.
sam2/__init__.py - Local module · Small external wrapper used to adapt or integrate the exported model.
src/Module.py
Connections 24
- Checkpoints → Export ONNX: weights · Export uses downloaded checkpoint files.
- Model configs → Build SAM2: selects · Model YAML selects the variant to assemble.
- Export ONNX → Build SAM2: invokes · Export entrypoint calls model construction.
- Build SAM2 → SAM2 base: creates · Builder instantiates the core SAM2 model.
- SAM2 base → Image encoder: uses · Core model wires in image feature extraction.
- SAM2 base → Prompt encoder: uses · Core model wires in prompt encoding.
- SAM2 base → Mask decoder: uses · Core model wires in mask decoding.
- SAM2 base → Memory encoder: uses · Video-capable model includes memory encoding.
- SAM2 base → Memory attention: uses · Video-capable model includes memory attention.
- Image encoder → Hiera backbone: built on · The encoder is implemented with the Hiera backbone.
- Image encoder → Position encoding: adds · Spatial encoding supports image features.
- Prompt encoder → Transformer blocks: runs through · Prompt features are processed by transformer blocks.
- Mask decoder → Transformer blocks: uses · Decoder relies on transformer internals.
- Image predictor → SAM2 base: calls · Image predictor runs the assembled SAM2 model.
- Image predictor → Utils: preprocesses · Image path depends on transforms and helpers.
- Auto mask gen → Image predictor: builds on · Automatic mask generation uses the image predictor path.
- Auto mask gen → Connected comps: postprocesses · Mask generation can rely on native connected-component acceleration.
- Video predictor → SAM2 base: calls · Video predictor runs the same core model with temporal state.
- Video predictor → Memory encoder: stores · Video path encodes past outputs into memory.
- Video predictor → Memory attention: attends · Video path retrieves memory when processing later frames.
- Local module → Export ONNX: adapts · Local wrapper likely bridges the model into export or integration flow.
- SAM2 package → Build SAM2: exposes · Package surface makes the implementation importable for assembly.
- SAM2 package → Image predictor: exposes · Package surface exposes image inference APIs.
- SAM2 package → Video predictor: exposes · Package surface exposes video inference APIs.