# kaai520/skeleton-guided-mamba architecture

> Architecture diagram of the GitHub repository kaai520/skeleton-guided-mamba, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kaai520/skeleton-guided-mamba
- Repository: https://github.com/kaai520/skeleton-guided-mamba

## Overview

SG-Mamba localizes short swallowing-related micro-actions in VFSS videos by combining appearance features with skeleton heatmaps, then predicting and evaluating temporal action segments. The repository also includes two-stage and two-tower training/evaluation variants and reusable Mamba sequence-model components. The README documents the multimodal method; sampled code shows heatmap extraction and parts of evaluation and modeling, but not the full training/model wiring, so uncertain call links are omitted.

Main entry points: [`sg-mamba/libs/datasets/swallow.py`](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/datasets/swallow.py) (Temporal datasets), [`sg-mamba/libs/modeling/wrapper.py`](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/modeling/wrapper.py) (Model wrapper), [`sg-mamba/eval2stage.py`](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/eval2stage.py) (Two-stage evaluation), [`sg-mamba/eval.py`](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/eval.py) (Evaluation entrypoints)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Input features"]
  node_vfss["VFSS videos"]
  node_keypoints["Keypoint inference"]
  node_heatmaps["Skeleton heatmaps<br/>[extract_heatmap.py]"]
  node_appearance["I3D appearance features<br/>[pytorch_i3d.py]"]
  node_datasets["Temporal datasets<br/>[swallow.py]"]
end

subgraph group_model["Temporal modeling"]
  node_model_wrap["Model wrapper<br/>[wrapper.py]"]
  node_meta_arch["Localization architecture<br/>[meta_archs.py]"]
  node_backbones["Video backbones<br/>[backbones.py]"]
  node_blocks["Cross-modal blocks<br/>[blocks.py]"]
  node_losses["Localization losses<br/>[losses.py]"]
end

subgraph group_workflow["Training and evaluation"]
  node_train["Training entrypoints<br/>[train2tower.py]"]
  node_train_stage["Two-stage training<br/>[train2stage.py]"]
  node_train_ddp["Distributed training<br/>[train2tower_ddp.py]"]
  node_eval_two["Two-stage evaluation<br/>[eval2stage.py]"]
  node_eval_main["Evaluation entrypoints<br/>[eval.py]"]
  node_postprocess["Segment postprocessing<br/>[postprocessing.py]"]
  node_metrics["Localization metrics<br/>[metrics.py]"]
end

subgraph group_mamba["Mamba runtime"]
  node_mamba_modules["Mamba mixers<br/>[mamba_v2m.py]"]
  node_scan_api["Selective scan API"]
  node_scan_cuda["Scan CUDA kernels<br/>[selective_scan.cpp]"]
  node_conv_cuda["Causal convolution<br/>[causal_conv1d.cpp]"]
end

node_researcher(("Researcher"))
node_results["Temporal detections"]

node_researcher -->|"provides videos"| node_vfss
node_vfss -->|"infers landmarks"| node_keypoints
node_keypoints -->|"encodes skeleton"| node_heatmaps
node_researcher -->|"starts training"| node_train
node_researcher -->|"starts training"| node_train_stage
node_researcher -->|"starts training"| node_train_ddp
node_researcher -->|"starts evaluation"| node_eval_two
node_researcher -->|"starts evaluation"| node_eval_main
node_appearance -->|"provides features"| node_datasets
node_heatmaps -->|"provides modality"| node_datasets
node_datasets -->|"supplies examples"| node_train
node_datasets -->|"supplies examples"| node_train_stage
node_datasets -->|"supplies examples"| node_train_ddp
node_train -->|"optimizes"| node_model_wrap
node_train_stage -->|"optimizes"| node_model_wrap
node_train_ddp -->|"optimizes"| node_model_wrap
node_model_wrap -->|"dispatches"| node_meta_arch
node_meta_arch -->|"extracts features"| node_backbones
node_meta_arch -->|"fuses modalities"| node_blocks
node_meta_arch -->|"computes loss"| node_losses
node_blocks -->|"uses mixers"| node_mamba_modules
node_mamba_modules -->|"runs scan"| node_scan_api
node_scan_api -->|"dispatches kernels"| node_scan_cuda
node_scan_api -->|"runs convolution"| node_conv_cuda
node_eval_two -->|"loads examples"| node_datasets
node_eval_two -->|"runs inference"| node_model_wrap
node_eval_two -->|"refines segments"| node_postprocess
node_eval_two -->|"evaluates detections"| node_metrics
node_eval_main -->|"loads examples"| node_datasets
node_eval_main -->|"runs inference"| node_model_wrap
node_eval_main -->|"refines segments"| node_postprocess
node_eval_main -->|"evaluates detections"| node_metrics
node_postprocess -->|"produces segments"| node_results

click node_keypoints "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/utils/inference_keypoints_api.py"
click node_heatmaps "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/tools/extract_heatmap.py"
click node_appearance "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/pytorch_i3d.py"
click node_datasets "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/datasets/swallow.py"
click node_train "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/train2tower.py"
click node_train_stage "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/train2stage.py"
click node_train_ddp "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/train2tower_ddp.py"
click node_model_wrap "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/modeling/wrapper.py"
click node_meta_arch "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/modeling/meta_archs.py"
click node_backbones "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/modeling/backbones.py"
click node_blocks "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/modeling/blocks.py"
click node_losses "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/modeling/losses.py"
click node_mamba_modules "https://github.com/kaai520/skeleton-guided-mamba/blob/main/mamba/mamba_ssm/modules/mamba_v2m.py"
click node_scan_api "https://github.com/kaai520/skeleton-guided-mamba/blob/main/mamba/mamba_ssm/ops/selective_scan_interface.py"
click node_scan_cuda "https://github.com/kaai520/skeleton-guided-mamba/blob/main/mamba/csrc/selective_scan/selective_scan.cpp"
click node_conv_cuda "https://github.com/kaai520/skeleton-guided-mamba/blob/main/causal-conv1d/csrc/causal_conv1d.cpp"
click node_eval_two "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/eval2stage.py"
click node_eval_main "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/eval.py"
click node_postprocess "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/utils/postprocessing.py"
click node_metrics "https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/utils/metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_vfss,node_keypoints,node_heatmaps,node_appearance,node_datasets toneBlue
class node_model_wrap,node_meta_arch,node_backbones,node_blocks,node_losses toneAmber
class node_train,node_train_stage,node_train_ddp,node_eval_two,node_eval_main,node_postprocess,node_metrics toneMint
class node_mamba_modules,node_scan_api,node_scan_cuda,node_conv_cuda toneRose
class node_researcher,node_results toneIndigo
```

## Components

### Input features

- **VFSS videos**
- [Keypoint inference](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/utils/inference_keypoints_api.py): `sg-mamba/libs/utils/inference_keypoints_api.py`
- [Skeleton heatmaps](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/tools/extract_heatmap.py): `sg-mamba/tools/extract_heatmap.py`
- [I3D appearance features](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/pytorch_i3d.py): `sg-mamba/pytorch_i3d.py`
- [Temporal datasets](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/datasets/swallow.py): `sg-mamba/libs/datasets/swallow.py`

### Temporal modeling

- [Model wrapper](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/modeling/wrapper.py): `sg-mamba/libs/modeling/wrapper.py`
- [Localization architecture](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/modeling/meta_archs.py): `sg-mamba/libs/modeling/meta_archs.py`
- [Video backbones](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/modeling/backbones.py): `sg-mamba/libs/modeling/backbones.py`
- [Cross-modal blocks](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/modeling/blocks.py): `sg-mamba/libs/modeling/blocks.py`
- [Localization losses](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/modeling/losses.py): `sg-mamba/libs/modeling/losses.py`

### Training and evaluation

- [Training entrypoints](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/train2tower.py): `sg-mamba/train2tower.py`
- [Two-stage training](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/train2stage.py): `sg-mamba/train2stage.py`
- [Distributed training](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/train2tower_ddp.py): `sg-mamba/train2tower_ddp.py`
- [Two-stage evaluation](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/eval2stage.py): `sg-mamba/eval2stage.py`
- [Evaluation entrypoints](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/eval.py): `sg-mamba/eval.py`
- [Segment postprocessing](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/utils/postprocessing.py): `sg-mamba/libs/utils/postprocessing.py`
- [Localization metrics](https://github.com/kaai520/skeleton-guided-mamba/blob/main/sg-mamba/libs/utils/metrics.py): `sg-mamba/libs/utils/metrics.py`

### Mamba runtime

- [Mamba mixers](https://github.com/kaai520/skeleton-guided-mamba/blob/main/mamba/mamba_ssm/modules/mamba_v2m.py): `mamba/mamba_ssm/modules/mamba_v2m.py`
- [Selective scan API](https://github.com/kaai520/skeleton-guided-mamba/blob/main/mamba/mamba_ssm/ops/selective_scan_interface.py): `mamba/mamba_ssm/ops/selective_scan_interface.py`
- [Scan CUDA kernels](https://github.com/kaai520/skeleton-guided-mamba/blob/main/mamba/csrc/selective_scan/selective_scan.cpp): `mamba/csrc/selective_scan/selective_scan.cpp`
- [Causal convolution](https://github.com/kaai520/skeleton-guided-mamba/blob/main/causal-conv1d/csrc/causal_conv1d.cpp): `causal-conv1d/csrc/causal_conv1d.cpp`

### Other components

- **Researcher**
- **Temporal detections**

## Connections

- Researcher → VFSS videos: provides videos
- VFSS videos → Keypoint inference: infers landmarks
- Keypoint inference → Skeleton heatmaps: encodes skeleton
- Researcher → Training entrypoints: starts training
- Researcher → Two-stage training: starts training
- Researcher → Distributed training: starts training
- Researcher → Two-stage evaluation: starts evaluation
- Researcher → Evaluation entrypoints: starts evaluation
- I3D appearance features → Temporal datasets: provides features
- Skeleton heatmaps → Temporal datasets: provides modality
- Temporal datasets → Training entrypoints: supplies examples
- Temporal datasets → Two-stage training: supplies examples
- Temporal datasets → Distributed training: supplies examples
- Training entrypoints → Model wrapper: optimizes
- Two-stage training → Model wrapper: optimizes
- Distributed training → Model wrapper: optimizes
- Model wrapper → Localization architecture: dispatches
- Localization architecture → Video backbones: extracts features
- Localization architecture → Cross-modal blocks: fuses modalities
- Localization architecture → Localization losses: computes loss
- Cross-modal blocks → Mamba mixers: uses mixers
- Mamba mixers → Selective scan API: runs scan
- Selective scan API → Scan CUDA kernels: dispatches kernels
- Selective scan API → Causal convolution: runs convolution
- Two-stage evaluation → Temporal datasets: loads examples
- Two-stage evaluation → Model wrapper: runs inference
- Two-stage evaluation → Segment postprocessing: refines segments
- Two-stage evaluation → Localization metrics: evaluates detections
- Evaluation entrypoints → Temporal datasets: loads examples
- Evaluation entrypoints → Model wrapper: runs inference
- Evaluation entrypoints → Segment postprocessing: refines segments
- Evaluation entrypoints → Localization metrics: evaluates detections
- Segment postprocessing → Temporal detections: produces segments