# oguzaybilir/locateanything-3b-finetuning architecture

> Architecture diagram of the GitHub repository oguzaybilir/locateanything-3b-finetuning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/oguzaybilir/locateanything-3b-finetuning
- Repository: https://github.com/oguzaybilir/locateanything-3b-finetuning

## Overview

This repository combines NVIDIA’s LocateAnything vision-language grounding model with a community LoRA fine-tuning and inference workflow. Users can turn LabelMe annotations into SFT data, fine-tune the checkpoint, run image detection or a Gradio demo, and compare and visualize evaluation results. The model also supports slow autoregressive, fast PBD, and hybrid decoding; the fork adds batched hybrid inference with sparse range attention. The README documents workflow stages; internal model and batch links below are limited to relationships visible in excerpts or the source index.

Main entry points: [`modeling_locateanything.py`](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/modeling_locateanything.py) (Grounding Model), [`modeling_qwen2.py`](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/modeling_qwen2.py) (Qwen2 Language Model), [`batch_utils/hybrid_runtime.py`](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/batch_utils/hybrid_runtime.py) (Batch Model Runtime), [`batch_infer.py`](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/batch_infer.py) (Batch Inference API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Fine-tuning Workflow"]
  node_dataset["SFT Dataset Builder<br/>[build_dataset.py]"]
  node_trainer["LoRA Trainer<br/>[train_lora.py]"]
end

subgraph group_serving["Inference and Evaluation"]
  node_predict["Detection CLI<br/>[predict.py]"]
  node_demo["Gradio Demo<br/>[app.py]"]
  node_evaluate["Mode Comparison<br/>[eval_compare.py]"]
  node_visualize["Eval Visualizer<br/>[visualize_eval.py]"]
end

subgraph group_model["Model Runtime"]
  node_processor["Image and Prompt Processing"]
  node_grounding["Grounding Model"]
  node_vision["MoonViT Vision Encoder<br/>[modeling_vit.py]"]
  node_language["Qwen2 Language Model<br/>[modeling_qwen2.py]"]
  node_generation["Generation Modes<br/>[generate_utils.py]"]
  node_mask_sdpa["SDPA Mask Utilities<br/>[mask_sdpa_utils.py]"]
  node_mask_magi["Magi Mask Utilities<br/>[mask_magi_utils.py]"]
  node_config_la["LocateAnything Config"]
  node_config_qwen["Qwen2 Config"]
end

subgraph group_batch["Batched Decoding"]
  node_batch_entry["Batch Inference API<br/>[batch_infer.py]"]
  node_hybrid["Hybrid Batch Engine<br/>[engine_hybrid.py]"]
  node_runtime["Batch Model Runtime<br/>[hybrid_runtime.py]"]
  node_range["Sparse Range Attention<br/>[range_attention.py]"]
end

node_user(("User"))

node_user -->|"builds data"| node_dataset
node_dataset -->|"provides SFT data"| node_trainer
node_user -->|"fine-tunes model"| node_trainer
node_user -->|"runs detection"| node_predict
node_user -->|"opens demo"| node_demo
node_user -->|"compares modes"| node_evaluate
node_evaluate -.->|"writes evaluation results"| node_visualize
node_grounding -->|"uses config"| node_config_la
node_grounding -->|"uses vision encoder"| node_vision
node_grounding -->|"uses language model"| node_language
node_grounding -->|"uses generation helpers"| node_generation
node_grounding -->|"imports masks"| node_mask_sdpa
node_grounding -->|"imports masks"| node_mask_magi
node_language -->|"uses config"| node_config_qwen
node_language -->|"imports masks"| node_mask_sdpa
node_language -->|"imports masks"| node_mask_magi
node_batch_entry -->|"dispatches to"| node_hybrid
node_batch_entry -->|"loads through"| node_runtime
node_hybrid -->|"uses runtime"| node_runtime
node_runtime -->|"uses attention"| node_range

click node_dataset "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/build_dataset.py"
click node_trainer "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/train_lora.py"
click node_predict "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/predict.py"
click node_demo "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/app.py"
click node_evaluate "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/eval_compare.py"
click node_visualize "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/visualize_eval.py"
click node_processor "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/processing_locateanything.py"
click node_grounding "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/modeling_locateanything.py"
click node_vision "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/modeling_vit.py"
click node_language "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/modeling_qwen2.py"
click node_generation "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/generate_utils.py"
click node_mask_sdpa "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/mask_sdpa_utils.py"
click node_mask_magi "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/mask_magi_utils.py"
click node_config_la "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/configuration_locateanything.py"
click node_config_qwen "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/configuration_qwen2.py"
click node_batch_entry "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/batch_infer.py"
click node_hybrid "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/batch_utils/engine_hybrid.py"
click node_runtime "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/batch_utils/hybrid_runtime.py"
click node_range "https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/kernel_utils/range_attention.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset,node_trainer,node_user toneBlue
class node_predict,node_demo,node_evaluate,node_visualize toneAmber
class node_processor,node_grounding,node_vision,node_language,node_generation,node_mask_sdpa,node_mask_magi,node_config_la,node_config_qwen toneMint
class node_batch_entry,node_hybrid,node_runtime,node_range toneRose
```

## Components

### Fine-tuning Workflow

- [SFT Dataset Builder](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/build_dataset.py): `build_dataset.py`
- [LoRA Trainer](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/train_lora.py): `train_lora.py`

### Inference and Evaluation

- [Detection CLI](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/predict.py): `predict.py`
- [Gradio Demo](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/app.py): `app.py`
- [Mode Comparison](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/eval_compare.py): `eval_compare.py`
- [Eval Visualizer](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/visualize_eval.py): `visualize_eval.py`

### Model Runtime

- [Image and Prompt Processing](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/processing_locateanything.py): `processing_locateanything.py`
- [Grounding Model](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/modeling_locateanything.py): `modeling_locateanything.py`
- [MoonViT Vision Encoder](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/modeling_vit.py): `modeling_vit.py`
- [Qwen2 Language Model](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/modeling_qwen2.py): `modeling_qwen2.py`
- [Generation Modes](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/generate_utils.py): `generate_utils.py`
- [SDPA Mask Utilities](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/mask_sdpa_utils.py): `mask_sdpa_utils.py`
- [Magi Mask Utilities](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/mask_magi_utils.py): `mask_magi_utils.py`
- [LocateAnything Config](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/configuration_locateanything.py): `configuration_locateanything.py`
- [Qwen2 Config](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/configuration_qwen2.py): `configuration_qwen2.py`

### Batched Decoding

- [Batch Inference API](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/batch_infer.py): `batch_infer.py`
- [Hybrid Batch Engine](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/batch_utils/engine_hybrid.py): `batch_utils/engine_hybrid.py`
- [Batch Model Runtime](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/batch_utils/hybrid_runtime.py): `batch_utils/hybrid_runtime.py`
- [Sparse Range Attention](https://github.com/oguzaybilir/locateanything-3b-finetuning/blob/main/kernel_utils/range_attention.py): `kernel_utils/range_attention.py`

### Other components

- **User**

## Connections

- User → SFT Dataset Builder: builds data (evidence: `README.md`)
- SFT Dataset Builder → LoRA Trainer: provides SFT data (evidence: `README.md`)
- User → LoRA Trainer: fine-tunes model (evidence: `README.md`)
- User → Detection CLI: runs detection (evidence: `README.md`)
- User → Gradio Demo: opens demo (evidence: `README.md`)
- User → Mode Comparison: compares modes (evidence: `README.md`)
- Mode Comparison → Eval Visualizer: writes evaluation results
- Grounding Model → LocateAnything Config: uses config (evidence: `modeling_locateanything.py`)
- Grounding Model → MoonViT Vision Encoder: uses vision encoder (evidence: `modeling_locateanything.py`)
- Grounding Model → Qwen2 Language Model: uses language model (evidence: `modeling_locateanything.py`)
- Grounding Model → Generation Modes: uses generation helpers (evidence: `modeling_locateanything.py`)
- Grounding Model → SDPA Mask Utilities: imports masks (evidence: `modeling_locateanything.py`)
- Grounding Model → Magi Mask Utilities: imports masks (evidence: `modeling_locateanything.py`)
- Qwen2 Language Model → Qwen2 Config: uses config (evidence: `modeling_qwen2.py`)
- Qwen2 Language Model → SDPA Mask Utilities: imports masks (evidence: `modeling_qwen2.py`)
- Qwen2 Language Model → Magi Mask Utilities: imports masks (evidence: `modeling_qwen2.py`)
- Batch Inference API → Hybrid Batch Engine: dispatches to (evidence: `batch_infer.py`)
- Batch Inference API → Batch Model Runtime: loads through (evidence: `batch_infer.py`)
- Hybrid Batch Engine → Batch Model Runtime: uses runtime (evidence: `batch_utils/engine_hybrid.py`)
- Batch Model Runtime → Sparse Range Attention: uses attention (evidence: `batch_utils/hybrid_runtime.py`)
