How oceanflowlab/quatrope works
QuatRoPE is a PyTorch 3D scene-language model for object grounding and spatial reasoning. Its main workflow prepares scene annotations and features, loads paired scene/question data, combines object and visual features with 3D positions, and generates answers through a LLaMA-based model. Training and evaluation cover grounding and language benchmarks; the README also describes ground-truth and Mask3D segmentation stages and an ASR benchmark. The graph reflects documented capabilities, but several links among unsampled model and preprocessing modules are intentionally omitted.
Main entry points: train.py Train And Infer · graph3dllm.py Scene-Language Model · prepare_mask3d_data.py Scene Preparation · base_dataset.py Scene Dataset
Components 19
Scene Data
- Scene Annotations
- Scene Preparation
preprocess/prepare_mask3d_data.py - Task Annotations
preprocess/prepare_objalign_annos.py - Scene Dataset
dataset/base_dataset.py - Data Loaders
dataset/dataloader.py
Spatial Model
- Scene-Language Model
models/graph3dllm.py - 3D Position Encoding
models/position_embedding.py - LLaMA Backbone
models/modeling_llama.py - MoE LoRA
models/moe/layer.py - Task Prompts
prompts/prompts.py
Training Runtime
- Train And Infer
tasks/train.py - Generation Support
generation/utils.py - Model Checkpoint
Evaluation
- Benchmark Inputs
ASR/gen_asr.py - Task Metrics
utils/eval.py - Evaluation Results
Other components
- Researcher
- 3D Scene Files
- Pretrained LLaMA
Connections 21
- Researcher → Train And Infer: starts run
- Scene Annotations → Scene Preparation: provides scenes
- 3D Scene Files → Scene Preparation: provides geometry
- Scene Preparation → Scene Dataset: prepares features
- Scene Annotations → Task Annotations: provides labels
- Task Annotations → Scene Dataset: prepares annotations
- Train And Infer → Data Loaders: sets up
- Data Loaders → Scene Dataset: loads batches
- Train And Infer → Scene-Language Model: trains or evaluates
- Scene-Language Model → 3D Position Encoding: encodes positions
- Scene-Language Model → LLaMA Backbone: uses backbone
- Pretrained LLaMA → LLaMA Backbone: initializes weights
- Scene-Language Model → MoE LoRA: uses adapters
- Task Prompts → Scene-Language Model: guides tasks
- Scene-Language Model → Generation Support: generates answers
- Train And Infer → Task Metrics: scores predictions
- Benchmark Inputs → Train And Infer: supplies ASR data
- Train And Infer → Model Checkpoint: saves weights
- Train And Infer → Evaluation Results: reports metrics
- Task Metrics → Evaluation Results: computes scores
- Researcher → Evaluation Results: reviews