How oguzaybilir/locateanything-3b-finetuning works
This repository combines NVIDIA’s LocateAnything vision-language grounding model with a community LoRA fine-tuning and inference workflow. Users can turn LabelMe annotations into SFT data, fine-tune the checkpoint, run image detection or a Gradio demo, and compare and visualize evaluation results. The model also supports slow autoregressive, fast PBD, and hybrid decoding; the fork adds batched hybrid inference with sparse range attention. The README documents workflow stages; internal model and batch links below are limited to relationships visible in excerpts or the source index.
Main entry points: modeling_locateanything.py Grounding Model · modeling_qwen2.py Qwen2 Language Model · hybrid_runtime.py Batch Model Runtime · batch_infer.py Batch Inference API
Components 20
Fine-tuning Workflow
- SFT Dataset Builder
build_dataset.py - LoRA Trainer
train_lora.py
Inference and Evaluation
- Detection CLI
predict.py - Gradio Demo
app.py - Mode Comparison
eval_compare.py - Eval Visualizer
visualize_eval.py
Model Runtime
- Image and Prompt Processing
processing_locateanything.py - Grounding Model
modeling_locateanything.py - MoonViT Vision Encoder
modeling_vit.py - Qwen2 Language Model
modeling_qwen2.py - Generation Modes
generate_utils.py - SDPA Mask Utilities
mask_sdpa_utils.py - Magi Mask Utilities
mask_magi_utils.py - LocateAnything Config
configuration_locateanything.py - Qwen2 Config
configuration_qwen2.py
Batched Decoding
- Batch Inference API
batch_infer.py - Hybrid Batch Engine
batch_utils/engine_hybrid.py - Batch Model Runtime
batch_utils/hybrid_runtime.py - Sparse Range Attention
kernel_utils/range_attention.py
Other components
- User
Connections 20
- User → SFT Dataset Builder: builds data
README.md - SFT Dataset Builder → LoRA Trainer: provides SFT data
README.md - User → LoRA Trainer: fine-tunes model
README.md - User → Detection CLI: runs detection
README.md - User → Gradio Demo: opens demo
README.md - User → Mode Comparison: compares modes
README.md - Mode Comparison → Eval Visualizer: writes evaluation results
- Grounding Model → LocateAnything Config: uses config
modeling_locateanything.py - Grounding Model → MoonViT Vision Encoder: uses vision encoder
modeling_locateanything.py - Grounding Model → Qwen2 Language Model: uses language model
modeling_locateanything.py - Grounding Model → Generation Modes: uses generation helpers
modeling_locateanything.py - Grounding Model → SDPA Mask Utilities: imports masks
modeling_locateanything.py - Grounding Model → Magi Mask Utilities: imports masks
modeling_locateanything.py - Qwen2 Language Model → Qwen2 Config: uses config
modeling_qwen2.py - Qwen2 Language Model → SDPA Mask Utilities: imports masks
modeling_qwen2.py - Qwen2 Language Model → Magi Mask Utilities: imports masks
modeling_qwen2.py - Batch Inference API → Hybrid Batch Engine: dispatches to
batch_infer.py - Batch Inference API → Batch Model Runtime: loads through
batch_infer.py - Hybrid Batch Engine → Batch Model Runtime: uses runtime
batch_utils/engine_hybrid.py - Batch Model Runtime → Sparse Range Attention: uses attention
batch_utils/hybrid_runtime.py