How linukc/beyondbarequeries works
BBQ builds an open-vocabulary 3D scene representation from RGB-D sequences, then answers natural-language object-grounding queries against the saved scene. The mapping workflow loads a dataset, integrates frames into object detections, postprocesses and projects the map, describes objects, and saves object and node outputs; grounding selects candidate objects and uses spatial relations to choose a referent. The README also documents evaluation on NR3D, SR3D, and ScanRefer. Mapping internals are represented from the tree and module names; their wiring was not confirmed in the sampled source.
Main entry points: main.py Mapping runner · query.py Interactive query · datasets.py RGB-D datasets · llm_interface.py Llama3 grounding
Components 21
Scene mapping
- Mapping runner
main.py - RGB-D datasets
bbq/datasets/datasets.py - Nodes constructor
bbq/objects_map/nodes_constructor.py - Detection assembly
bbq/objects_map/detections_assembler.py - Object association
bbq/objects_map/objects_associator.py - View projection
bbq/objects_map/projector.py - Object description
bbq/objects_map/describer.py - Map structures
bbq/objects_map/utils
Visual perception
- DINO features
bbq/models/dino.py - LLaVA model
bbq/models/llava.py - Mask generation
bbq/models/masks
Object grounding
- Interactive query
query.py - Llama3 grounding
bbq/grounding/llm_interface.py - Spatial relations
bbq/grounding/utils.py
Grounding evaluation
- Grounding evaluation
evaluation/object_grounding
Other components
- Query user
- RGB-D data
- Saved scene
- Local LLM weights
- NR3D SR3D ScanRefer
- OpenAI batch API
Connections 12
- RGB-D data → RGB-D datasets: reads frames
- Mapping runner → RGB-D datasets: loads dataset
- Mapping runner → Nodes constructor: integrates and processes
- Mapping runner → RGB-D datasets: reads poses and intrinsics
- Mapping runner → Saved scene: writes scene outputs
- Query user → Interactive query: submits query
- Interactive query → Llama3 grounding: selects candidates and referent
- Interactive query → Saved scene: loads scene
- Interactive query → Local LLM weights: loads weights
- Llama3 grounding → Spatial relations: computes spatial relations
- NR3D SR3D ScanRefer → Grounding evaluation: provides evaluation data
- Grounding evaluation → OpenAI batch API: uses batch inference