How kaai520/skeleton-guided-mamba works
SG-Mamba localizes short swallowing-related micro-actions in VFSS videos by combining appearance features with skeleton heatmaps, then predicting and evaluating temporal action segments. The repository also includes two-stage and two-tower training/evaluation variants and reusable Mamba sequence-model components. The README documents the multimodal method; sampled code shows heatmap extraction and parts of evaluation and modeling, but not the full training/model wiring, so uncertain call links are omitted.
Main entry points: swallow.py Temporal datasets · wrapper.py Model wrapper · eval2stage.py Two-stage evaluation · eval.py Evaluation entrypoints
Components 23
Input features
- VFSS videos
- Keypoint inference
sg-mamba/libs/utils/inference_keypoints_api.py - Skeleton heatmaps
sg-mamba/tools/extract_heatmap.py - I3D appearance features
sg-mamba/pytorch_i3d.py - Temporal datasets
sg-mamba/libs/datasets/swallow.py
Temporal modeling
- Model wrapper
sg-mamba/libs/modeling/wrapper.py - Localization architecture
sg-mamba/libs/modeling/meta_archs.py - Video backbones
sg-mamba/libs/modeling/backbones.py - Cross-modal blocks
sg-mamba/libs/modeling/blocks.py - Localization losses
sg-mamba/libs/modeling/losses.py
Training and evaluation
- Training entrypoints
sg-mamba/train2tower.py - Two-stage training
sg-mamba/train2stage.py - Distributed training
sg-mamba/train2tower_ddp.py - Two-stage evaluation
sg-mamba/eval2stage.py - Evaluation entrypoints
sg-mamba/eval.py - Segment postprocessing
sg-mamba/libs/utils/postprocessing.py - Localization metrics
sg-mamba/libs/utils/metrics.py
Mamba runtime
- Mamba mixers
mamba/mamba_ssm/modules/mamba_v2m.py - Selective scan API
mamba/mamba_ssm/ops/selective_scan_interface.py - Scan CUDA kernels
mamba/csrc/selective_scan/selective_scan.cpp - Causal convolution
causal-conv1d/csrc/causal_conv1d.cpp
Other components
- Researcher
- Temporal detections
Connections 33
- Researcher → VFSS videos: provides videos
- VFSS videos → Keypoint inference: infers landmarks
- Keypoint inference → Skeleton heatmaps: encodes skeleton
- Researcher → Training entrypoints: starts training
- Researcher → Two-stage training: starts training
- Researcher → Distributed training: starts training
- Researcher → Two-stage evaluation: starts evaluation
- Researcher → Evaluation entrypoints: starts evaluation
- I3D appearance features → Temporal datasets: provides features
- Skeleton heatmaps → Temporal datasets: provides modality
- Temporal datasets → Training entrypoints: supplies examples
- Temporal datasets → Two-stage training: supplies examples
- Temporal datasets → Distributed training: supplies examples
- Training entrypoints → Model wrapper: optimizes
- Two-stage training → Model wrapper: optimizes
- Distributed training → Model wrapper: optimizes
- Model wrapper → Localization architecture: dispatches
- Localization architecture → Video backbones: extracts features
- Localization architecture → Cross-modal blocks: fuses modalities
- Localization architecture → Localization losses: computes loss
- Cross-modal blocks → Mamba mixers: uses mixers
- Mamba mixers → Selective scan API: runs scan
- Selective scan API → Scan CUDA kernels: dispatches kernels
- Selective scan API → Causal convolution: runs convolution
- Two-stage evaluation → Temporal datasets: loads examples
- Two-stage evaluation → Model wrapper: runs inference
- Two-stage evaluation → Segment postprocessing: refines segments
- Two-stage evaluation → Localization metrics: evaluates detections
- Evaluation entrypoints → Temporal datasets: loads examples
- Evaluation entrypoints → Model wrapper: runs inference
- Evaluation entrypoints → Segment postprocessing: refines segments
- Evaluation entrypoints → Localization metrics: evaluates detections
- Segment postprocessing → Temporal detections: produces segments