How wngkj/lang2segtrack works
Language-to-Segment&Track turns text or interactive prompts into image masks or persistent video tracks. The video workflow accepts frames and prompts, uses Grounding-DINO for text-based detections and SAM2 for prompted segmentation and temporal propagation, then supports backward tracking and continuous category detection in separate script variants. The README documents these capabilities; source-index references and excerpts support the wrapper and model relationships shown. Some script and model internals were not sampled, so uncertain implementation links are omitted.
Main entry points: sam2_base.py SAM2 core · sam.py SAM wrapper · build_sam.py Model builder · sam2_utils.py Model utilities
Components 27
User workflows
- Image and video
scripts/lang2segtrack.py - Backward tracking
scripts/lang2segtrack_with_backward.py - Continuous detection
scripts/lang2segtrack_keep_detection.py - Video processing
scripts/process_video.py
Text detection
- Grounding-DINO
models/gdino/models/gdino.py - Detection utilities
models/gdino/utils.py
SAM2 segmentation
- SAM wrapper
models/sam2/sam.py - Model builder
models/sam2/sam2/build_sam.py - Image predictor
models/sam2/sam2/sam2_image_predictor.py - Automatic masks
models/sam2/sam2/automatic_mask_generator.py - Video predictor
models/sam2/sam2/sam2_video_predictor.py - SAM2 core
models/sam2/sam2/modeling/sam2_base.py - Image backbone
models/sam2/sam2/modeling/backbones/hieradet.py - Prompt encoder
models/sam2/sam2/modeling/sam/prompt_encoder.py - Mask decoder
models/sam2/sam2/modeling/sam/mask_decoder.py - SAM transformer
models/sam2/sam2/modeling/sam/transformer.py - Memory encoder
models/sam2/sam2/modeling/memory_encoder.py - Memory attention
models/sam2/sam2/modeling/memory_attention.py - Position encoding
models/sam2/sam2/modeling/position_encoding.py - Model utilities
models/sam2/sam2/modeling/sam2_utils.py - Mask utilities
models/sam2/sam2/utils/amg.py
Tracking support
- Frame loading
models/sam2/sam2/utils/misc.py - Kalman filter
models/sam2/sam2/utils/kalman_filter.py - Object registry
utils/ObjectInfoManager.py - Mask and video tools
utils/utils.py - Track colors
utils/color.py
Other components
- User
Connections 46
- User → Image and video: runs
- User → Backward tracking: runs
- User → Continuous detection: runs
- User → Video processing: runs
- Image and video → Grounding-DINO: uses
scripts/lang2segtrack.py - Image and video → SAM wrapper: uses
scripts/lang2segtrack.py - Image and video → Mask and video tools: uses
scripts/lang2segtrack.py - Image and video → Track colors: uses
scripts/lang2segtrack.py - Backward tracking → Grounding-DINO: uses
scripts/lang2segtrack_with_backward.py - Backward tracking → SAM wrapper: uses
scripts/lang2segtrack_with_backward.py - Backward tracking → Mask and video tools: uses
scripts/lang2segtrack_with_backward.py - Continuous detection → Grounding-DINO: uses
scripts/lang2segtrack_keep_detection.py - Continuous detection → SAM wrapper: uses
scripts/lang2segtrack_keep_detection.py - Continuous detection → Object registry: uses
scripts/lang2segtrack_keep_detection.py - Continuous detection → Track colors: uses
scripts/lang2segtrack_keep_detection.py - Video processing → Model builder: uses
scripts/process_video.py - Video processing → Mask and video tools: uses
scripts/process_video.py - Grounding-DINO → Detection utilities: imports
models/gdino/models/gdino.py - SAM wrapper → Model builder: builds
models/sam2/sam.py - SAM wrapper → Image predictor: creates
models/sam2/sam.py - SAM wrapper → Automatic masks: creates
models/sam2/sam.py - SAM wrapper → Video predictor: creates
models/sam2/sam.py - Model builder → SAM2 core: instantiates
models/sam2/sam2/build_sam.py - Image predictor → SAM2 core: uses
models/sam2/sam2/sam2_image_predictor.py - Image predictor → Model builder: loads via
models/sam2/sam2/sam2_image_predictor.py - Automatic masks → Image predictor: uses
models/sam2/sam2/automatic_mask_generator.py - Automatic masks → SAM2 core: uses
models/sam2/sam2/automatic_mask_generator.py - Automatic masks → Mask utilities: uses
models/sam2/sam2/automatic_mask_generator.py - Automatic masks → Model builder: imports
models/sam2/sam2/automatic_mask_generator.py - Video predictor → SAM2 core: uses
models/sam2/sam2/sam2_video_predictor.py - Video predictor → Frame loading: uses
models/sam2/sam2/sam2_video_predictor.py - Video predictor → Model builder: imports
models/sam2/sam2/sam2_video_predictor.py - SAM2 core → Image backbone: uses
- SAM2 core → Prompt encoder: uses
models/sam2/sam2/modeling/sam2_base.py - SAM2 core → Mask decoder: uses
models/sam2/sam2/modeling/sam2_base.py - SAM2 core → Kalman filter: uses
models/sam2/sam2/modeling/sam2_base.py - Prompt encoder → Position encoding: uses
models/sam2/sam2/modeling/sam/prompt_encoder.py - Prompt encoder → Model utilities: uses
models/sam2/sam2/modeling/sam/prompt_encoder.py - Mask decoder → Model utilities: uses
models/sam2/sam2/modeling/sam/mask_decoder.py - SAM transformer → Position encoding: uses
models/sam2/sam2/modeling/sam/transformer.py - SAM transformer → Model utilities: uses
models/sam2/sam2/modeling/sam/transformer.py - Memory attention → SAM transformer: uses
models/sam2/sam2/modeling/memory_attention.py - Memory attention → Model utilities: uses
models/sam2/sam2/modeling/memory_attention.py - Memory encoder → Model utilities: uses
models/sam2/sam2/modeling/memory_encoder.py - Model utilities → Frame loading: uses
models/sam2/sam2/modeling/sam2_utils.py - Mask and video tools → Track colors: imports
utils/utils.py