# magic-research/sa2va architecture

> Architecture diagram of the GitHub repository magic-research/sa2va, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/magic-research/sa2va
- Repository: https://github.com/magic-research/sa2va

## Overview

##### Project type and purpose
- This is not a web app or database-backed service but a research-oriented machine learning monorepo.
- Its goal is to build “Sa2VA,” a unified multi-modal model that marries SAM-2 (segment-anything video) with LLaVA (vision-language LLM) for dense grounded understanding of images & videos.
- It also hosts related subprojects (e.g., GLAMM, Pixel-SAIL) and provides training, evaluation, and demo tooling.

##### High-level file-structure analysis
Top-level folders:
- demo/            → Jupyter notebook + simple inference scripts + model-local Gradio demo
- projects/        → three subprojects:
– glamm/       → datasets, dataset utilities, glamm model for grounded language‐and‐mask tasks
– llava_sam2/  → configs, dataset loaders, model definitions, training scripts, HF conversion, Gradio frontend
– pixel_sail/  → (separate README, future Pixel-SAIL code)
- third_parts/     → vendored external libraries (mmdet, SAM2 implementation, ReVOS, etc.)
- tools/           → shell scripts to launch distributed training, evaluation, testing
- vlm/             → general engine: runner loops, hooks, utilities for video & vision-language model training
- assets/, requirements.txt, README.md, LICENSE

##### README insights
- Architecture unifies three roles:
1) Dataset ingestion & collation
2) Model stacking: SAM2 backbones + LLaVA chat LLM
3) Training & inference engines (Deepspeed, mmcv, custom runner loops)
- Provides pre-trained model dependencies & paths to HuggingFace collections.
- Demo: Gradio app + scripts to predict on image folders or videos.
- Training: uses tools/dist.sh, 8 GPUs recommended, outputs checkpoints convertible to HF.

##### Main components & their interactions
a. Data layer
- Data sources: multiple datasets (Video-Datasets: DAVIS, MeViS, ReVOS; Ref-seg: RefCOCO, etc.; GLAMM data; Osprey; LLaVA chat & pretrain)
- Data folder structure (data/) + projects/<subproject>/datasets modules + third_parts/* utilities
- Collate / encode functions prepare batches of (images, masks, prompts) for model
b. Model layer
- SAM2 backbone (third_parts/sam2) for mask encoding/prompt decoding
- LLaVA / InternVL LLM connectors (projects/llava_sam2/hf/models & models/ extension + tokenizer)
- GLAMM model variants under projects/glamm for segmentation-&-caption tasks
- Pixel-SAIL code (future extension)
c. Training & engine layer
- vlm/engine: hooks (dataset_info), runner loops (train, eval, video loops)
- tools/dist.sh & tools/train.py launch multi-GPU training (Deepspeed Zero2)
- Configuration files in projects/llava_sam2/configs (.py) select model sizes & training hyperparams
d. Evaluation & metrics
- projects/llava_sam2/evaluation: script wrappers, metrics for RefCOCO, GCG, region caption, video-VOS
- tools/eval/* entrypoints for dataset-specific evaluation using model checkpoints
e. Inference / Demo
- demo/ scripts & notebooks
- projects/llava_sam2/gradio/app.py + app_utils.py for interactive chat & segmentation via HTTP
- third_parts/sam2 predictors for single image/video mask generation
f. External integrations
- HuggingFace (model download & HF conversion)
- Slurm / HPC (tools/slurm.sh)
- Jupyter (demo.ipynb)
- deepspeed, mmcv, transformers, peft

##### Architectural patterns & design principles
- Modular dataset‐model‐engine separation (like PyTorch Lightning)
- Vendored third-party code under third_parts to ensure reproducibility
- Config‐driven experiments for easy hyperparameter & model‐size swaps
- Single entrypoints for training (tools/dist.sh), evaluation (tools/eval/*), inference (demo/)
- Clear subfolder per subproject (glamm, llava_sam2, pixel_sail)

## Diagram (Mermaid)

```mermaid
flowchart TD
    DataSources["Data Sources:<br/>• Video Datasets (DAVIS, MeViS, ReVOS, ChatUniVi)<br/>• Ref-Seg (RefCOCO, RefCLEF, GCG)<br/>• GLAMM data, Osprey, LLaVA"]:::data
    DataLoaderPreproc["Data Loader & Preprocessor:<br/>projects/glamm/datasets/<br/>projects/llava_sam2/datasets/<br/>collect_fns.py<br/>encode_fn.py<br/>video_io.py<br/>third_parts/mmdet/datasets/<br/>third_parts/revos/utils/"]:::process
    ModelStack["Model Stack:<br/>third_parts/sam2/modeling/<br/>third_parts/sam2/sam2_configs/<br/>sam2_image_predictor.py<br/>sam2_video_predictor.py<br/>projects/llava_sam2/models/llava_sam2.py<br/>projects/llava_sam2/models/extension/sam2_base.py<br/>projects/llava_sam2/models/internvl.py<br/>projects/llava_sam2/models/lisa.py<br/>projects/glamm/models/glamm.py<br/>projects/glamm/models/region_encoder.py<br/>projects/llava_sam2/hf/models/"]:::model
    TrainingEngine["Training Engine:<br/>vlm/engine/runner/loops.py<br/>vlm/engine/runner/video_loops.py<br/>vlm/engine/hooks/dataset_info_hook.py<br/>tools/train.py<br/>tools/dist.sh<br/>tools/slurm.sh<br/>projects/llava_sam2/configs/sa2va_*.py"]:::process
    Checkpoint["Checkpoint Storage:<br/>./work-dir, ./pretrained<br/>convert_to_hf.py<br/>convert_to_whole.py"]:::storage
    Evaluation["Evaluation:<br/>projects/llava_sam2/evaluation/<br/>tools/eval/eval_*.py"]:::process
    Inference["Inference & Demo:<br/>demo.ipynb<br/>demo/predict-img.py<br/>demo/predict-vid.py<br/>projects/llava_sam2/gradio/app.py<br/>projects/llava_sam2/gradio/app_utils.py"]:::process
    HF["HuggingFace Model Hub"]:::external
    Slurm["Slurm/HPC Scheduler"]:::external
    Jupyter["Jupyter Notebook"]:::external
    DLlib["DeepSpeed / mmcv / Transformers / PEFT"]:::external

    DataSources -->|"(image, mask, prompt) batches"| DataLoaderPreproc
    DataLoaderPreproc --> ModelStack
    ModelStack --> TrainingEngine
    TrainingEngine --> Checkpoint
    Checkpoint --> Evaluation
    Checkpoint --> Inference

    HF <-->|"model download & conversion"| Checkpoint
    HF --> ModelStack
    Slurm --> TrainingEngine
    Jupyter --> Inference
    DLlib --> TrainingEngine

    click DataLoaderPreproc "https://github.com/magic-research/sa2va/tree/main/projects/glamm/datasets/"
    click DataLoaderPreproc "https://github.com/magic-research/sa2va/tree/main/projects/llava_sam2/datasets/"
    click DataLoaderPreproc "https://github.com/magic-research/sa2va/blob/main/projects/llava_sam2/datasets/collect_fns.py"
    click DataLoaderPreproc "https://github.com/magic-research/sa2va/blob/main/projects/llava_sam2/datasets/encode_fn.py"
    click DataLoaderPreproc "https://github.com/magic-research/sa2va/blob/main/video_io.py"
    click DataLoaderPreproc "https://github.com/magic-research/sa2va/tree/main/third_parts/mmdet/datasets/"
    click DataLoaderPreproc "https://github.com/magic-research/sa2va/tree/main/third_parts/revos/utils/"

    click ModelStack "https://github.com/magic-research/sa2va/tree/main/third_parts/sam2/modeling/"
    click ModelStack "https://github.com/magic-research/sa2va/tree/main/third_parts/sam2/sam2_configs/"
    click ModelStack "https://github.com/magic-research/sa2va/blob/main/third_parts/sam2/sam2_image_predictor.py"
    click ModelStack "https://github.com/magic-research/sa2va/blob/main/third_parts/sam2/sam2_video_predictor.py"
    click ModelStack "https://github.com/magic-research/sa2va/blob/main/projects/llava_sam2/models/llava_sam2.py"
    click ModelStack "https://github.com/magic-research/sa2va/blob/main/projects/llava_sam2/models/extension/sam2_base.py"
    click ModelStack "https://github.com/magic-research/sa2va/blob/main/projects/llava_sam2/models/internvl.py"
    click ModelStack "https://github.com/magic-research/sa2va/blob/main/projects/llava_sam2/models/lisa.py"
    click ModelStack "https://github.com/magic-research/sa2va/blob/main/projects/glamm/models/glamm.py"
    click ModelStack "https://github.com/magic-research/sa2va/blob/main/projects/glamm/models/region_encoder.py"
    click ModelStack "https://github.com/magic-research/sa2va/tree/main/projects/llava_sam2/hf/models/"

    click TrainingEngine "https://github.com/magic-research/sa2va/blob/main/vlm/engine/runner/loops.py"
    click TrainingEngine "https://github.com/magic-research/sa2va/blob/main/vlm/engine/runner/video_loops.py"
    click TrainingEngine "https://github.com/magic-research/sa2va/blob/main/vlm/engine/hooks/dataset_info_hook.py"
    click TrainingEngine "https://github.com/magic-research/sa2va/blob/main/tools/train.py"
    click TrainingEngine "https://github.com/magic-research/sa2va/blob/main/tools/dist.sh"
    click TrainingEngine "https://github.com/magic-research/sa2va/blob/main/tools/slurm.sh"
    click TrainingEngine "https://github.com/magic-research/sa2va/tree/main/projects/llava_sam2/configs/"

    click Checkpoint "https://github.com/magic-research/sa2va/blob/main/projects/llava_sam2/hf/convert_to_hf.py"
    click Checkpoint "https://github.com/magic-research/sa2va/blob/main/projects/llava_sam2/hf/convert_to_whole.py"

    click Evaluation "https://github.com/magic-research/sa2va/tree/main/projects/llava_sam2/evaluation/"
    click Evaluation "https://github.com/magic-research/sa2va/tree/main/tools/eval/"

    click Inference "https://github.com/magic-research/sa2va/blob/main/demo.ipynb"
    click Inference "https://github.com/magic-research/sa2va/blob/main/demo/predict-img.py"
    click Inference "https://github.com/magic-research/sa2va/blob/main/demo/predict-vid.py"
    click Inference "https://github.com/magic-research/sa2va/blob/main/projects/llava_sam2/gradio/app.py"
    click Inference "https://github.com/magic-research/sa2va/blob/main/projects/llava_sam2/gradio/app_utils.py"

    classDef data fill:#BBDEFB,stroke:#000
    classDef process fill:#C8E6C9,stroke:#000
    classDef model fill:#FFE0B2,stroke:#000
    classDef storage fill:#B3E5FC,stroke:#000
    classDef external fill:#E1BEE7,stroke:#000
```
