# thippeswammy/sam2 architecture

> Architecture diagram of the GitHub repository thippeswammy/sam2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thippeswammy/sam2
- Repository: https://github.com/thippeswammy/sam2

## Overview

##### Project Type and Purpose
- This is an open‐source, Python-based video processing pipeline built around the SAM2 segmentation model. Its goal is to automate frame extraction, interactive annotation, mask prediction, overlay generation, verification, and final video compilation.

##### High-Level File Structure
- Top-level “sam2/” and “sam2_configs/”: the core SAM2 model code and YAML configs.
- “segment-anything-3/”: the main CLI pipeline (sam2_video_predictor.py and supporting modules).
- “DataVisualizationEditingTool/”: a GUI tool for visual data inspection and editing.
- “DatasetManager/”: utilities to convert and export datasets (e.g., YOLO format).
- “sav_dataset/”: evaluation and visualization tooling for the SAV benchmark.
- Other folders (“code/”, “notebooks/”, “tools/”, “DataPoints/”, “NVIDIA/”, “lanes/”): examples, experiments, helpers, GPU checks, and raw data.

##### Key Insights from the README
- The pipeline is invoked via a single script (`sam2_video_predictor.py`) with flags for input range, batch size, directories, and FPS.
- Stages: frame extraction → interactive point annotation → batched mask prediction (GPU) → mask overlay → manual verification → video assembly.
- Relies on external dependencies: PyTorch, OpenCV, GPUtil, pygetwindow, and the SAM2 model checkpoint/config.

a. Main Components
- Input Layer
– VideoInputs directory (Video1.mp4, Video2.mp4, …)
- Frame Extraction Service
– Extracts frames to `working_dir/images/`
- Annotation GUI
– Live display; captures (x,y,label) points per frame
– Depends on pygetwindow/OpenCV
- Batch Processing Manager
– Groups extracted frames into batches (configurable via `--batch_size`)
- SAM2 Mask Predictor
– Loads model via sam2 library and `sam2_hiera_*.yaml`
– Outputs raw masks to `working_dir/render/`
- Mask Overlay Module
– Combines original frames + mask → `working_dir/overlap/`
- Verification Step
– Prompts user to accept/reject overlays
– Populates `verified_img_dir` and `verified_mask_dir`
- Video Composer
– Reads verified folders; writes three .mp4 outputs (Org, Mask, Overlay)
- Configuration & Checkpoints
– Global config files and model weights accessed by predictor

b. Data Flow & Interactions
– VideoInputs → FrameExtractor → Frames
– Frames + Annotation GUI → Annotated Points
– Annotated Points + Frames → Batch Processor → SAM2 Predictor
– Predictor → Masks → Overlay Module → Overlays
– Overlays + Original Frames → Verification UI → Verified Artifacts
– Verified Artifacts → VideoComposer → Final Videos

c. Architectural Patterns
– Modular pipeline: clearly separated stages (single-responsibility modules)
– Batch processing for GPU efficiency
– Interactive feedback loop (annotation + verification)

d. Technologies & Frameworks
– Python 3.8+
– PyTorch (GPU inference)
– OpenCV (I/O, GUI windows)
– GPUtil & tqdm (resource and progress management)
– sam2 library (model definition)
– YAML configs for model parameters

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Pipeline Entry
    CLI["sam2_video_predictor.py (segment-anything-3/)"]:::process

    %% Input Layer
    VideoInput["VideoInputs (User Directory)"]:::datastore

    %% Core Pipeline
    subgraph "Core Pipeline"
        FE["Frame Extraction Service"]:::process
        GUI["Annotation GUI"]:::interactive
        BPM["Batch Processing Manager"]:::process
        SMP["SAM2 Mask Predictor"]:::process
        MO["Mask Overlay Module"]:::process
        VER["Verification Step"]:::interactive
        VC["Video Composer"]:::process
    end

    %% Working Directories
    subgraph "Working Directory"
        IMG["working_dir/images/"]:::datastore
        RND["working_dir/render/"]:::datastore
        OVL["working_dir/overlap/"]:::datastore
        VIDF["verified_img_dir/"]:::datastore
        VIDM["verified_mask_dir/"]:::datastore
    end

    %% Configuration & Checkpoints
    subgraph "Configs & Checkpoints"
        CFG1["sam2_configs/*.yaml"]:::config
        CKPT["checkpoints/download_ckpts.sh"]:::config
    end

    %% Extensions
    subgraph "Extensions"
        DM["DatasetManager (YOLO Export)"]:::ext
        SAV["SAV Benchmark Tool"]:::ext
        NB["Notebooks & Code Examples"]:::ext
    end

    %% Main Flow
    VideoInput -->|Provides videos| CLI
    CLI -->|extract frames| FE
    FE -->|writes frames| IMG
    IMG -->|loads frames| GUI
    GUI -->|annotations| BPM
    IMG -->|frames| BPM
    BPM -->|batches| SMP
    SMP -->|raw masks| RND
    RND -->|overlays| MO
    MO -->|writes overlays| OVL
    OVL -->|review| VER
    IMG -->|original frames| VER
    VER -->|verified images| VIDF
    VER -->|verified masks| VIDM
    VIDF -->|compose videos| VC
    VIDM -->|compose masks| VC

    %% CLI uses configs
    CLI -->|loads| CFG1
    SMP -->|uses model| CFG1
    SMP -->|downloads weights| CKPT

    %% Extensions interactions
    GUI -.->|export YOLO| DM
    VER -.->|evaluate| SAV
    CLI -.->|examples| NB

    %% Click Events
    click FE "https://github.com/thippeswammy/sam2/blob/master/segment-anything-3/FrameExtractor.py"
    click FE "https://github.com/thippeswammy/sam2/blob/master/segment-anything-3/videos/multithreaded_video_frame_extractor.py"
    click GUI "https://github.com/thippeswammy/sam2/blob/master/DataVisualizationEditingTool/main.py"
    click GUI "https://github.com/thippeswammy/sam2/blob/master/DataVisualizationEditingTool/event_handler.py"
    click GUI "https://github.com/thippeswammy/sam2/blob/master/DataVisualizationEditingTool/plot_manager.py"
    click GUI "https://github.com/thippeswammy/sam2/blob/master/DataVisualizationEditingTool/data_loader.py"
    click GUI "https://github.com/thippeswammy/sam2/blob/master/DataVisualizationEditingTool/data_manager.py"
    click GUI "https://github.com/thippeswammy/sam2/blob/master/DataVisualizationEditingTool/curve_manager.py"
    click BPM "https://github.com/thippeswammy/sam2/blob/master/segment-anything-3/pipeline.py"
    click BPM "https://github.com/thippeswammy/sam2/blob/master/segment-anything-3/pipeline2.py"
    click SMP "https://github.com/thippeswammy/sam2/blob/master/segment-anything-3/sam2_video_predictor.py"
    click SMP "https://github.com/thippeswammy/sam2/blob/master/sam2/sam2_video_predictor.py"
    click SMP "https://github.com/thippeswammy/sam2/blob/master/sam2/automatic_mask_generator.py"
    click SMP "https://github.com/thippeswammy/sam2/blob/master/sam2/build_sam.py"
    click SMP "https://github.com/thippeswammy/sam2/blob/master/sam2_configs/sam2_hiera_b+.yaml"
    click SMP "https://github.com/thippeswammy/sam2/blob/master/sam2_configs/sam2_hiera_l.yaml"
    click SMP "https://github.com/thippeswammy/sam2/blob/master/sam2_configs/sam2_hiera_s.yaml"
    click SMP "https://github.com/thippeswammy/sam2/blob/master/sam2_configs/sam2_hiera_t.yaml"
    click MO "https://github.com/thippeswammy/sam2/blob/master/segment-anything-3/ImageOverlayProcessor.py"
    click MO "https://github.com/thippeswammy/sam2/blob/master/segment-anything-3/videos/ImageOverlayProcessor.py"
    click VER "https://github.com/thippeswammy/sam2/blob/master/segment-anything-3/pipeline2.py"
    click VC "https://github.com/thippeswammy/sam2/blob/master/segment-anything-3/VideoCreator.py"
    click VC "https://github.com/thippeswammy/sam2/blob/master/segment-anything-3/videos/imgs2video.py"
    click DM "https://github.com/thippeswammy/sam2/blob/master/DatasetManager/YolovDatasetManager/create_yolo_structure.py"
    click DM "https://github.com/thippeswammy/sam2/blob/master/DatasetManager/YolovDatasetManager/dataset_manager1.py"
    click DM "https://github.com/thippeswammy/sam2/blob/master/DatasetManager/YolovDatasetManager/DatasetCreatere.py"
    click SAV "https://github.com/thippeswammy/sam2/blob/master/sav_dataset/sav_evaluator.py"
    click SAV "https://github.com/thippeswammy/sam2/blob/master/sav_dataset/utils/sav_benchmark.py"
    click SAV "https://github.com/thippeswammy/sam2/blob/master/sav_dataset/utils/sav_utils.py"
    click NB "https://github.com/thippeswammy/sam2/blob/master/notebooks/automatic_mask_generator_example.ipynb"
    click NB "https://github.com/thippeswammy/sam2/blob/master/notebooks/image_predictor_example.ipynb"
    click NB "https://github.com/thippeswammy/sam2/blob/master/notebooks/video_predictor_example.ipynb"
    click NB "https://github.com/thippeswammy/sam2/blob/master/code/video_frame_processor26.py"
    click NB "https://github.com/thippeswammy/sam2/blob/master/code/video_predictor_example.py"

    %% Styles
    classDef process fill:#D6EAF8,stroke:#2E86C1,stroke-width:1px
    classDef datastore fill:#D5F5E3,stroke:#27AE60,stroke-width:1px
    classDef interactive fill:#FCF3CF,stroke:#F1C40F,stroke-width:1px,stroke-dasharray: 5 5
    classDef config fill:#FCF0F3,stroke:#C0392B,stroke-width:1px,stroke-dasharray: 2 2
    classDef ext fill:#EAECEE,stroke:#ABB2B9,stroke-width:1px,stroke-dasharray: 4 2
```
