# wngkj/lang2segtrack architecture

> Architecture diagram of the GitHub repository wngkj/lang2segtrack, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wngkj/lang2segtrack
- Repository: https://github.com/wngkj/lang2segtrack

## Overview

Language-to-Segment&Track turns text or interactive prompts into image masks or persistent video tracks. The video workflow accepts frames and prompts, uses Grounding-DINO for text-based detections and SAM2 for prompted segmentation and temporal propagation, then supports backward tracking and continuous category detection in separate script variants. The README documents these capabilities; source-index references and excerpts support the wrapper and model relationships shown. Some script and model internals were not sampled, so uncertain implementation links are omitted.

Main entry points: [`models/sam2/sam2/modeling/sam2_base.py`](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam2_base.py) (SAM2 core), [`models/sam2/sam.py`](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam.py) (SAM wrapper), [`models/sam2/sam2/build_sam.py`](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/build_sam.py) (Model builder), [`models/sam2/sam2/modeling/sam2_utils.py`](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam2_utils.py) (Model utilities)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User workflows"]
  node_standard["Image and video<br/>[lang2segtrack.py]"]
  node_backward["Backward tracking"]
  node_continuous["Continuous detection"]
  node_video_process["Video processing<br/>[process_video.py]"]
end

subgraph group_detection["Text detection"]
  node_gdino["Grounding-DINO<br/>[gdino.py]"]
  node_gdino_utils["Detection utilities<br/>[utils.py]"]
end

subgraph group_sam["SAM2 segmentation"]
  node_sam_wrapper["SAM wrapper<br/>[sam.py]"]
  node_sam_build["Model builder<br/>[build_sam.py]"]
  node_image_predictor["Image predictor"]
  node_mask_generator["Automatic masks"]
  node_video_predictor["Video predictor"]
  node_sam_base["SAM2 core<br/>[sam2_base.py]"]
  node_backbone["Image backbone<br/>[hieradet.py]"]
  node_prompt_encoder["Prompt encoder<br/>[prompt_encoder.py]"]
  node_mask_decoder["Mask decoder<br/>[mask_decoder.py]"]
  node_transformer["SAM transformer<br/>[transformer.py]"]
  node_memory_encoder["Memory encoder<br/>[memory_encoder.py]"]
  node_memory_attention["Memory attention"]
  node_position_encoding["Position encoding"]
  node_sam_utils["Model utilities<br/>[sam2_utils.py]"]
  node_amg["Mask utilities<br/>[amg.py]"]
end

subgraph group_support["Tracking support"]
  node_video_frames["Frame loading<br/>[misc.py]"]
  node_kalman["Kalman filter<br/>[kalman_filter.py]"]
  node_object_info["Object registry"]
  node_app_utils["Mask and video tools<br/>[utils.py]"]
  node_colors["Track colors<br/>[color.py]"]
end

node_user(("User"))

node_user -->|"runs"| node_standard
node_user -->|"runs"| node_backward
node_user -->|"runs"| node_continuous
node_user -->|"runs"| node_video_process
node_standard -->|"uses"| node_gdino
node_standard -->|"uses"| node_sam_wrapper
node_standard -->|"uses"| node_app_utils
node_standard -->|"uses"| node_colors
node_backward -->|"uses"| node_gdino
node_backward -->|"uses"| node_sam_wrapper
node_backward -->|"uses"| node_app_utils
node_continuous -->|"uses"| node_gdino
node_continuous -->|"uses"| node_sam_wrapper
node_continuous -->|"uses"| node_object_info
node_continuous -->|"uses"| node_colors
node_video_process -->|"uses"| node_sam_build
node_video_process -->|"uses"| node_app_utils
node_gdino -->|"imports"| node_gdino_utils
node_sam_wrapper -->|"builds"| node_sam_build
node_sam_wrapper -->|"creates"| node_image_predictor
node_sam_wrapper -->|"creates"| node_mask_generator
node_sam_wrapper -->|"creates"| node_video_predictor
node_sam_build -->|"instantiates"| node_sam_base
node_image_predictor -->|"uses"| node_sam_base
node_image_predictor -->|"loads via"| node_sam_build
node_mask_generator -->|"uses"| node_image_predictor
node_mask_generator -->|"uses"| node_sam_base
node_mask_generator -->|"uses"| node_amg
node_mask_generator -->|"imports"| node_sam_build
node_video_predictor -->|"uses"| node_sam_base
node_video_predictor -->|"uses"| node_video_frames
node_video_predictor -->|"imports"| node_sam_build
node_sam_base -.->|"uses"| node_backbone
node_sam_base -->|"uses"| node_prompt_encoder
node_sam_base -->|"uses"| node_mask_decoder
node_sam_base -->|"uses"| node_kalman
node_prompt_encoder -->|"uses"| node_position_encoding
node_prompt_encoder -->|"uses"| node_sam_utils
node_mask_decoder -->|"uses"| node_sam_utils
node_transformer -->|"uses"| node_position_encoding
node_transformer -->|"uses"| node_sam_utils
node_memory_attention -->|"uses"| node_transformer
node_memory_attention -->|"uses"| node_sam_utils
node_memory_encoder -->|"uses"| node_sam_utils
node_sam_utils -->|"uses"| node_video_frames
node_app_utils -->|"imports"| node_colors

click node_standard "https://github.com/wngkj/lang2segtrack/blob/main/scripts/lang2segtrack.py"
click node_backward "https://github.com/wngkj/lang2segtrack/blob/main/scripts/lang2segtrack_with_backward.py"
click node_continuous "https://github.com/wngkj/lang2segtrack/blob/main/scripts/lang2segtrack_keep_detection.py"
click node_video_process "https://github.com/wngkj/lang2segtrack/blob/main/scripts/process_video.py"
click node_gdino "https://github.com/wngkj/lang2segtrack/blob/main/models/gdino/models/gdino.py"
click node_gdino_utils "https://github.com/wngkj/lang2segtrack/blob/main/models/gdino/utils.py"
click node_sam_wrapper "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam.py"
click node_sam_build "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/build_sam.py"
click node_image_predictor "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/sam2_image_predictor.py"
click node_mask_generator "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/automatic_mask_generator.py"
click node_video_predictor "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/sam2_video_predictor.py"
click node_sam_base "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam2_base.py"
click node_backbone "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/backbones/hieradet.py"
click node_prompt_encoder "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam/prompt_encoder.py"
click node_mask_decoder "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam/mask_decoder.py"
click node_transformer "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam/transformer.py"
click node_memory_encoder "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/memory_encoder.py"
click node_memory_attention "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/memory_attention.py"
click node_position_encoding "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/position_encoding.py"
click node_sam_utils "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam2_utils.py"
click node_amg "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/utils/amg.py"
click node_video_frames "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/utils/misc.py"
click node_kalman "https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/utils/kalman_filter.py"
click node_object_info "https://github.com/wngkj/lang2segtrack/blob/main/utils/ObjectInfoManager.py"
click node_app_utils "https://github.com/wngkj/lang2segtrack/blob/main/utils/utils.py"
click node_colors "https://github.com/wngkj/lang2segtrack/blob/main/utils/color.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_standard,node_backward,node_continuous,node_video_process,node_user toneBlue
class node_gdino,node_gdino_utils toneAmber
class node_sam_wrapper,node_sam_build,node_image_predictor,node_mask_generator,node_video_predictor,node_sam_base,node_backbone,node_prompt_encoder,node_mask_decoder,node_transformer,node_memory_encoder,node_memory_attention,node_position_encoding,node_sam_utils,node_amg toneMint
class node_video_frames,node_kalman,node_object_info,node_app_utils,node_colors toneRose
```

## Components

### User workflows

- [Image and video](https://github.com/wngkj/lang2segtrack/blob/main/scripts/lang2segtrack.py): `scripts/lang2segtrack.py`
- [Backward tracking](https://github.com/wngkj/lang2segtrack/blob/main/scripts/lang2segtrack_with_backward.py): `scripts/lang2segtrack_with_backward.py`
- [Continuous detection](https://github.com/wngkj/lang2segtrack/blob/main/scripts/lang2segtrack_keep_detection.py): `scripts/lang2segtrack_keep_detection.py`
- [Video processing](https://github.com/wngkj/lang2segtrack/blob/main/scripts/process_video.py): `scripts/process_video.py`

### Text detection

- [Grounding-DINO](https://github.com/wngkj/lang2segtrack/blob/main/models/gdino/models/gdino.py): `models/gdino/models/gdino.py`
- [Detection utilities](https://github.com/wngkj/lang2segtrack/blob/main/models/gdino/utils.py): `models/gdino/utils.py`

### SAM2 segmentation

- [SAM wrapper](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam.py): `models/sam2/sam.py`
- [Model builder](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/build_sam.py): `models/sam2/sam2/build_sam.py`
- [Image predictor](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/sam2_image_predictor.py): `models/sam2/sam2/sam2_image_predictor.py`
- [Automatic masks](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/automatic_mask_generator.py): `models/sam2/sam2/automatic_mask_generator.py`
- [Video predictor](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/sam2_video_predictor.py): `models/sam2/sam2/sam2_video_predictor.py`
- [SAM2 core](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam2_base.py): `models/sam2/sam2/modeling/sam2_base.py`
- [Image backbone](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/backbones/hieradet.py): `models/sam2/sam2/modeling/backbones/hieradet.py`
- [Prompt encoder](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam/prompt_encoder.py): `models/sam2/sam2/modeling/sam/prompt_encoder.py`
- [Mask decoder](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam/mask_decoder.py): `models/sam2/sam2/modeling/sam/mask_decoder.py`
- [SAM transformer](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam/transformer.py): `models/sam2/sam2/modeling/sam/transformer.py`
- [Memory encoder](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/memory_encoder.py): `models/sam2/sam2/modeling/memory_encoder.py`
- [Memory attention](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/memory_attention.py): `models/sam2/sam2/modeling/memory_attention.py`
- [Position encoding](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/position_encoding.py): `models/sam2/sam2/modeling/position_encoding.py`
- [Model utilities](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/modeling/sam2_utils.py): `models/sam2/sam2/modeling/sam2_utils.py`
- [Mask utilities](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/utils/amg.py): `models/sam2/sam2/utils/amg.py`

### Tracking support

- [Frame loading](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/utils/misc.py): `models/sam2/sam2/utils/misc.py`
- [Kalman filter](https://github.com/wngkj/lang2segtrack/blob/main/models/sam2/sam2/utils/kalman_filter.py): `models/sam2/sam2/utils/kalman_filter.py`
- [Object registry](https://github.com/wngkj/lang2segtrack/blob/main/utils/ObjectInfoManager.py): `utils/ObjectInfoManager.py`
- [Mask and video tools](https://github.com/wngkj/lang2segtrack/blob/main/utils/utils.py): `utils/utils.py`
- [Track colors](https://github.com/wngkj/lang2segtrack/blob/main/utils/color.py): `utils/color.py`

### Other components

- **User**

## Connections

- User → Image and video: runs
- User → Backward tracking: runs
- User → Continuous detection: runs
- User → Video processing: runs
- Image and video → Grounding-DINO: uses (evidence: `scripts/lang2segtrack.py`)
- Image and video → SAM wrapper: uses (evidence: `scripts/lang2segtrack.py`)
- Image and video → Mask and video tools: uses (evidence: `scripts/lang2segtrack.py`)
- Image and video → Track colors: uses (evidence: `scripts/lang2segtrack.py`)
- Backward tracking → Grounding-DINO: uses (evidence: `scripts/lang2segtrack_with_backward.py`)
- Backward tracking → SAM wrapper: uses (evidence: `scripts/lang2segtrack_with_backward.py`)
- Backward tracking → Mask and video tools: uses (evidence: `scripts/lang2segtrack_with_backward.py`)
- Continuous detection → Grounding-DINO: uses (evidence: `scripts/lang2segtrack_keep_detection.py`)
- Continuous detection → SAM wrapper: uses (evidence: `scripts/lang2segtrack_keep_detection.py`)
- Continuous detection → Object registry: uses (evidence: `scripts/lang2segtrack_keep_detection.py`)
- Continuous detection → Track colors: uses (evidence: `scripts/lang2segtrack_keep_detection.py`)
- Video processing → Model builder: uses (evidence: `scripts/process_video.py`)
- Video processing → Mask and video tools: uses (evidence: `scripts/process_video.py`)
- Grounding-DINO → Detection utilities: imports (evidence: `models/gdino/models/gdino.py`)
- SAM wrapper → Model builder: builds (evidence: `models/sam2/sam.py`)
- SAM wrapper → Image predictor: creates (evidence: `models/sam2/sam.py`)
- SAM wrapper → Automatic masks: creates (evidence: `models/sam2/sam.py`)
- SAM wrapper → Video predictor: creates (evidence: `models/sam2/sam.py`)
- Model builder → SAM2 core: instantiates (evidence: `models/sam2/sam2/build_sam.py`)
- Image predictor → SAM2 core: uses (evidence: `models/sam2/sam2/sam2_image_predictor.py`)
- Image predictor → Model builder: loads via (evidence: `models/sam2/sam2/sam2_image_predictor.py`)
- Automatic masks → Image predictor: uses (evidence: `models/sam2/sam2/automatic_mask_generator.py`)
- Automatic masks → SAM2 core: uses (evidence: `models/sam2/sam2/automatic_mask_generator.py`)
- Automatic masks → Mask utilities: uses (evidence: `models/sam2/sam2/automatic_mask_generator.py`)
- Automatic masks → Model builder: imports (evidence: `models/sam2/sam2/automatic_mask_generator.py`)
- Video predictor → SAM2 core: uses (evidence: `models/sam2/sam2/sam2_video_predictor.py`)
- Video predictor → Frame loading: uses (evidence: `models/sam2/sam2/sam2_video_predictor.py`)
- Video predictor → Model builder: imports (evidence: `models/sam2/sam2/sam2_video_predictor.py`)
- SAM2 core → Image backbone: uses
- SAM2 core → Prompt encoder: uses (evidence: `models/sam2/sam2/modeling/sam2_base.py`)
- SAM2 core → Mask decoder: uses (evidence: `models/sam2/sam2/modeling/sam2_base.py`)
- SAM2 core → Kalman filter: uses (evidence: `models/sam2/sam2/modeling/sam2_base.py`)
- Prompt encoder → Position encoding: uses (evidence: `models/sam2/sam2/modeling/sam/prompt_encoder.py`)
- Prompt encoder → Model utilities: uses (evidence: `models/sam2/sam2/modeling/sam/prompt_encoder.py`)
- Mask decoder → Model utilities: uses (evidence: `models/sam2/sam2/modeling/sam/mask_decoder.py`)
- SAM transformer → Position encoding: uses (evidence: `models/sam2/sam2/modeling/sam/transformer.py`)
- SAM transformer → Model utilities: uses (evidence: `models/sam2/sam2/modeling/sam/transformer.py`)
- Memory attention → SAM transformer: uses (evidence: `models/sam2/sam2/modeling/memory_attention.py`)
- Memory attention → Model utilities: uses (evidence: `models/sam2/sam2/modeling/memory_attention.py`)
- Memory encoder → Model utilities: uses (evidence: `models/sam2/sam2/modeling/memory_encoder.py`)
- Model utilities → Frame loading: uses (evidence: `models/sam2/sam2/modeling/sam2_utils.py`)
- Mask and video tools → Track colors: imports (evidence: `utils/utils.py`)
