How magic-research/sa2va works
Project type and purpose
• This is not a web app or database-backed service but a research-oriented machine learning monorepo.
• Its goal is to build “Sa2VA,” a unified multi-modal model that marries SAM-2 (segment-anything video) with LLaVA (vision-language LLM) for dense grounded understanding of images & videos.
• It also hosts related subprojects (e.g., GLAMM, Pixel-SAIL) and provides training, evaluation, and demo tooling.
High-level file-structure analysis
Top-level folders:
• demo/ → Jupyter notebook + simple inference scripts + model-local Gradio demo
• projects/ → three subprojects:
– glamm/ → datasets, dataset utilities, glamm model for grounded language‐and‐mask tasks
– llava_sam2/ → configs, dataset loaders, model definitions, training scripts, HF conversion, Gradio frontend
– pixel_sail/ → (separate README, future Pixel-SAIL code)
• third_parts/ → vendored external libraries (mmdet, SAM2 implementation, ReVOS, etc.)
• tools/ → shell scripts to launch distributed training, evaluation, testing
• vlm/ → general engine: runner loops, hooks, utilities for video & vision-language model training
• assets/, requirements.txt, README.md, LICENSE
README insights
• Architecture unifies three roles:
1) Dataset ingestion & collation
2) Model stacking: SAM2 backbones + LLaVA chat LLM
3) Training & inference engines (Deepspeed, mmcv, custom runner loops)
• Provides pre-trained model dependencies & paths to HuggingFace collections.
• Demo: Gradio app + scripts to predict on image folders or videos.
• Training: uses tools/dist.sh, 8 GPUs recommended, outputs checkpoints convertible to HF.
Main components & their interactions
a. Data layer
• Data sources: multiple datasets (Video-Datasets: DAVIS, MeViS, ReVOS; Ref-seg: RefCOCO, etc.; GLAMM data; Osprey; LLaVA chat & pretrain)
• Data folder structure (data/) + projects/<subproject>/datasets modules + third_parts/* utilities
• Collate / encode functions prepare batches of (images, masks, prompts) for model
b. Model layer
• SAM2 backbone (third_parts/sam2) for mask encoding/prompt decoding
• LLaVA / InternVL LLM connectors (projects/llava_sam2/hf/models & models/ extension + tokenizer)
• GLAMM model variants under projects/glamm for segmentation-&-caption tasks
• Pixel-SAIL code (future extension)
c. Training & engine layer
• vlm/engine: hooks (dataset_info), runner loops (train, eval, video loops)
• tools/dist.sh & tools/train.py launch multi-GPU training (Deepspeed Zero2)
• Configuration files in projects/llava_sam2/configs (.py) select model sizes & training hyperparams
d. Evaluation & metrics
• projects/llava_sam2/evaluation: script wrappers, metrics for RefCOCO, GCG, region caption, video-VOS
• tools/eval/* entrypoints for dataset-specific evaluation using model checkpoints
e. Inference / Demo
• demo/ scripts & notebooks
• projects/llava_sam2/gradio/app.py + app_utils.py for interactive chat & segmentation via HTTP
• third_parts/sam2 predictors for single image/video mask generation
f. External integrations
• HuggingFace (model download & HF conversion)
• Slurm / HPC (tools/slurm.sh)
• Jupyter (demo.ipynb)
• deepspeed, mmcv, transformers, peft
Architectural patterns & design principles
• Modular dataset‐model‐engine separation (like PyTorch Lightning)
• Vendored third-party code under third_parts to ensure reproducibility
• Config‐driven experiments for easy hyperparameter & model‐size swaps
• Single entrypoints for training (tools/dist.sh), evaluation (tools/eval/*), inference (demo/)
• Clear subfolder per subproject (glamm, llava_sam2, pixel_sail)