How thippeswammy/sam2 works
Project Type and Purpose
• This is an open‐source, Python-based video processing pipeline built around the SAM2 segmentation model. Its goal is to automate frame extraction, interactive annotation, mask prediction, overlay generation, verification, and final video compilation.
High-Level File Structure
• Top-level “sam2/” and “sam2_configs/”: the core SAM2 model code and YAML configs.
• “segment-anything-3/”: the main CLI pipeline (sam2_video_predictor.py and supporting modules).
• “DataVisualizationEditingTool/”: a GUI tool for visual data inspection and editing.
• “DatasetManager/”: utilities to convert and export datasets (e.g., YOLO format).
• “sav_dataset/”: evaluation and visualization tooling for the SAV benchmark.
• Other folders (“code/”, “notebooks/”, “tools/”, “DataPoints/”, “NVIDIA/”, “lanes/”): examples, experiments, helpers, GPU checks, and raw data.
Key Insights from the README
• The pipeline is invoked via a single script (sam2_video_predictor.py) with flags for input range, batch size, directories, and FPS.
• Stages: frame extraction → interactive point annotation → batched mask prediction (GPU) → mask overlay → manual verification → video assembly.
• Relies on external dependencies: PyTorch, OpenCV, GPUtil, pygetwindow, and the SAM2 model checkpoint/config.
a. Main Components
• Input Layer
– VideoInputs directory (Video1.mp4, Video2.mp4, …)
• Frame Extraction Service
– Extracts frames to working_dir/images/
• Annotation GUI
– Live display; captures (x,y,label) points per frame
– Depends on pygetwindow/OpenCV
• Batch Processing Manager
– Groups extracted frames into batches (configurable via --batch_size)
• SAM2 Mask Predictor
– Loads model via sam2 library and sam2_hiera_*.yaml
– Outputs raw masks to working_dir/render/
• Mask Overlay Module
– Combines original frames + mask → working_dir/overlap/
• Verification Step
– Prompts user to accept/reject overlays
– Populates verified_img_dir and verified_mask_dir
• Video Composer
– Reads verified folders; writes three .mp4 outputs (Org, Mask, Overlay)
• Configuration & Checkpoints
– Global config files and model weights accessed by predictor
b. Data Flow & Interactions
– VideoInputs → FrameExtractor → Frames
– Frames + Annotation GUI → Annotated Points
– Annotated Points + Frames → Batch Processor → SAM2 Predictor
– Predictor → Masks → Overlay Module → Overlays
– Overlays + Original Frames → Verification UI → Verified Artifacts
– Verified Artifacts → VideoComposer → Final Videos
c. Architectural Patterns
– Modular pipeline: clearly separated stages (single-responsibility modules)
– Batch processing for GPU efficiency
– Interactive feedback loop (annotation + verification)
d. Technologies & Frameworks
– Python 3.8+
– PyTorch (GPU inference)
– OpenCV (I/O, GUI windows)
– GPUtil & tqdm (resource and progress management)
– sam2 library (model definition)
– YAML configs for model parameters