# linukc/beyondbarequeries architecture

> Architecture diagram of the GitHub repository linukc/beyondbarequeries, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/linukc/beyondbarequeries
- Repository: https://github.com/linukc/beyondbarequeries

## Overview

BBQ builds an open-vocabulary 3D scene representation from RGB-D sequences, then answers natural-language object-grounding queries against the saved scene. The mapping workflow loads a dataset, integrates frames into object detections, postprocesses and projects the map, describes objects, and saves object and node outputs; grounding selects candidate objects and uses spatial relations to choose a referent. The README also documents evaluation on NR3D, SR3D, and ScanRefer. Mapping internals are represented from the tree and module names; their wiring was not confirmed in the sampled source.

Main entry points: [`main.py`](https://github.com/linukc/beyondbarequeries/blob/main/main.py) (Mapping runner), [`query.py`](https://github.com/linukc/beyondbarequeries/blob/main/query.py) (Interactive query), [`bbq/datasets/datasets.py`](https://github.com/linukc/beyondbarequeries/blob/main/bbq/datasets/datasets.py) (RGB-D datasets), [`bbq/grounding/llm_interface.py`](https://github.com/linukc/beyondbarequeries/blob/main/bbq/grounding/llm_interface.py) (Llama3 grounding)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_mapping["Scene mapping"]
  node_main["Mapping runner<br/>[main.py]"]
  node_dataset["RGB-D datasets<br/>[datasets.py]"]
  node_constructor["Nodes constructor"]
  node_assembler["Detection assembly"]
  node_associator["Object association"]
  node_projector["View projection<br/>[projector.py]"]
  node_describer["Object description<br/>[describer.py]"]
  node_maputils["Map structures"]
end

subgraph group_perception["Visual perception"]
  node_dino["DINO features<br/>[dino.py]"]
  node_llava["LLaVA model<br/>[llava.py]"]
  node_masks["Mask generation"]
end

subgraph group_grounding["Object grounding"]
  node_query["Interactive query<br/>[query.py]"]
  node_llm["Llama3 grounding<br/>[llm_interface.py]"]
  node_relations["Spatial relations<br/>[utils.py]"]
end

subgraph group_evaluation["Grounding evaluation"]
  node_eval["Grounding evaluation"]
end

node_user(("Query user"))
node_rgbd["RGB-D data"]
node_scene["Saved scene"]
node_localmodel["Local LLM weights"]
node_benchmarks["NR3D SR3D ScanRefer"]
node_openai["OpenAI batch API"]

node_rgbd -->|"reads frames"| node_dataset
node_main -->|"loads dataset"| node_dataset
node_main -->|"integrates and processes"| node_constructor
node_main -->|"reads poses and intrinsics"| node_dataset
node_main -->|"writes scene outputs"| node_scene
node_user -->|"submits query"| node_query
node_query -->|"selects candidates and referent"| node_llm
node_query -->|"loads scene"| node_scene
node_query -->|"loads weights"| node_localmodel
node_llm -->|"computes spatial relations"| node_relations
node_benchmarks -.->|"provides evaluation data"| node_eval
node_eval -.->|"uses batch inference"| node_openai

click node_main "https://github.com/linukc/beyondbarequeries/blob/main/main.py"
click node_dataset "https://github.com/linukc/beyondbarequeries/blob/main/bbq/datasets/datasets.py"
click node_constructor "https://github.com/linukc/beyondbarequeries/blob/main/bbq/objects_map/nodes_constructor.py"
click node_assembler "https://github.com/linukc/beyondbarequeries/blob/main/bbq/objects_map/detections_assembler.py"
click node_associator "https://github.com/linukc/beyondbarequeries/blob/main/bbq/objects_map/objects_associator.py"
click node_projector "https://github.com/linukc/beyondbarequeries/blob/main/bbq/objects_map/projector.py"
click node_describer "https://github.com/linukc/beyondbarequeries/blob/main/bbq/objects_map/describer.py"
click node_maputils "https://github.com/linukc/beyondbarequeries/tree/main/bbq/objects_map/utils"
click node_dino "https://github.com/linukc/beyondbarequeries/blob/main/bbq/models/dino.py"
click node_llava "https://github.com/linukc/beyondbarequeries/blob/main/bbq/models/llava.py"
click node_masks "https://github.com/linukc/beyondbarequeries/tree/main/bbq/models/masks"
click node_query "https://github.com/linukc/beyondbarequeries/blob/main/query.py"
click node_llm "https://github.com/linukc/beyondbarequeries/blob/main/bbq/grounding/llm_interface.py"
click node_relations "https://github.com/linukc/beyondbarequeries/blob/main/bbq/grounding/utils.py"
click node_eval "https://github.com/linukc/beyondbarequeries/tree/main/evaluation/object_grounding"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_dataset,node_constructor,node_assembler,node_associator,node_projector,node_describer,node_maputils,node_user toneBlue
class node_dino,node_llava,node_masks toneAmber
class node_query,node_llm,node_relations,node_openai toneMint
class node_eval toneRose
class node_rgbd,node_scene,node_localmodel,node_benchmarks toneIndigo
```

## Components

### Scene mapping

- [Mapping runner](https://github.com/linukc/beyondbarequeries/blob/main/main.py): `main.py`
- [RGB-D datasets](https://github.com/linukc/beyondbarequeries/blob/main/bbq/datasets/datasets.py): `bbq/datasets/datasets.py`
- [Nodes constructor](https://github.com/linukc/beyondbarequeries/blob/main/bbq/objects_map/nodes_constructor.py): `bbq/objects_map/nodes_constructor.py`
- [Detection assembly](https://github.com/linukc/beyondbarequeries/blob/main/bbq/objects_map/detections_assembler.py): `bbq/objects_map/detections_assembler.py`
- [Object association](https://github.com/linukc/beyondbarequeries/blob/main/bbq/objects_map/objects_associator.py): `bbq/objects_map/objects_associator.py`
- [View projection](https://github.com/linukc/beyondbarequeries/blob/main/bbq/objects_map/projector.py): `bbq/objects_map/projector.py`
- [Object description](https://github.com/linukc/beyondbarequeries/blob/main/bbq/objects_map/describer.py): `bbq/objects_map/describer.py`
- [Map structures](https://github.com/linukc/beyondbarequeries/tree/main/bbq/objects_map/utils): `bbq/objects_map/utils`

### Visual perception

- [DINO features](https://github.com/linukc/beyondbarequeries/blob/main/bbq/models/dino.py): `bbq/models/dino.py`
- [LLaVA model](https://github.com/linukc/beyondbarequeries/blob/main/bbq/models/llava.py): `bbq/models/llava.py`
- [Mask generation](https://github.com/linukc/beyondbarequeries/tree/main/bbq/models/masks): `bbq/models/masks`

### Object grounding

- [Interactive query](https://github.com/linukc/beyondbarequeries/blob/main/query.py): `query.py`
- [Llama3 grounding](https://github.com/linukc/beyondbarequeries/blob/main/bbq/grounding/llm_interface.py): `bbq/grounding/llm_interface.py`
- [Spatial relations](https://github.com/linukc/beyondbarequeries/blob/main/bbq/grounding/utils.py): `bbq/grounding/utils.py`

### Grounding evaluation

- [Grounding evaluation](https://github.com/linukc/beyondbarequeries/tree/main/evaluation/object_grounding): `evaluation/object_grounding`

### Other components

- **Query user**
- **RGB-D data**
- **Saved scene**
- **Local LLM weights**
- **NR3D SR3D ScanRefer**
- **OpenAI batch API**

## Connections

- RGB-D data → RGB-D datasets: reads frames
- Mapping runner → RGB-D datasets: loads dataset
- Mapping runner → Nodes constructor: integrates and processes
- Mapping runner → RGB-D datasets: reads poses and intrinsics
- Mapping runner → Saved scene: writes scene outputs
- Query user → Interactive query: submits query
- Interactive query → Llama3 grounding: selects candidates and referent
- Interactive query → Saved scene: loads scene
- Interactive query → Local LLM weights: loads weights
- Llama3 grounding → Spatial relations: computes spatial relations
- NR3D SR3D ScanRefer → Grounding evaluation: provides evaluation data
- Grounding evaluation → OpenAI batch API: uses batch inference