# ardamamur/egoexor architecture

> Architecture diagram of the GitHub repository ardamamur/egoexor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ardamamur/egoexor
- Repository: https://github.com/ardamamur/egoexor

## Overview

EgoExOR is a multimodal operating-room dataset and scene-graph benchmark, with a baseline model for surgical activity understanding. The documented path runs from HDF5 data preparation and LLaVA-format sample generation through model training and inference, where predictions support scene-graph evaluation. The map also preserves the dataset’s visualization utilities and the baseline’s multimodal and language-model layers. Some subsystem internals are unsampled; links to them are omitted where the available excerpts, source index, or README do not establish a relationship.

Main entry points: [`scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py`](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py) (Scene graph model), [`scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py`](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py) (Sample formatter), [`scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/dataset_utils.py`](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/dataset_utils.py) (Graph label utilities), [`scene_graph_generation/LLaVA/llava/model/llava_arch.py`](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/model/llava_arch.py) (Multimodal integration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Dataset and preparation"]
  node_merge["HDF5 merger<br/>[merge_h5.py]"]
  node_loader["HDF5 reader<br/>[load_h5.py]"]
  node_formatgen["Sample formatter"]
  node_converters["Graph converters"]
  node_templates["Prompt templates"]
  node_datasetutils["Graph label utilities<br/>[dataset_utils.py]"]
end

subgraph group_model["Multimodal model"]
  node_builder["Model loader<br/>[builder.py]"]
  node_llavaarch["Multimodal integration<br/>[llava_arch.py]"]
  node_encoders["Modality encoders<br/>[builder.py]"]
  node_projector["Feature projectors<br/>[builder.py]"]
  node_pointprojector["Point-cloud projector"]
  node_conversation["Conversation formatting<br/>[conversation.py]"]
end

subgraph group_benchmark["Training and evaluation"]
  node_dataset["OR dataset<br/>[or_dataset.py]"]
  node_train["LLaVA trainer entry<br/>[train.py]"]
  node_trainer["Training engine<br/>[llava_trainer.py]"]
  node_sgmodel["Scene graph model"]
  node_transform["Input transformation"]
  node_infer["Inference and evaluation<br/>[main.py]"]
end

subgraph group_inspect["Data inspection"]
  node_visualize["Take visualization<br/>[visualize_take.py]"]
  node_timepoint["Timepoint visualization"]
end

node_researcher(("Dataset user"))
node_hdf5[("HDF5 dataset")]

node_researcher -->|"merges data"| node_merge
node_merge -->|"writes dataset"| node_hdf5
node_researcher -->|"generates samples"| node_formatgen
node_formatgen -->|"uses converters"| node_converters
node_formatgen -->|"uses templates"| node_templates
node_formatgen -->|"uses labels"| node_datasetutils
node_researcher -->|"starts training"| node_train
node_train -->|"uses trainer"| node_trainer
node_train -->|"uses labels"| node_datasetutils
node_researcher -->|"runs benchmark"| node_infer
node_infer -->|"loads evaluation data"| node_dataset
node_infer -->|"evaluates model"| node_sgmodel
node_sgmodel -->|"loads model"| node_builder
node_sgmodel -->|"transforms inputs"| node_transform
node_sgmodel -->|"uses dataset"| node_dataset
node_sgmodel -->|"uses labels"| node_datasetutils
node_dataset -->|"uses utilities"| node_datasetutils
node_builder -->|"builds model"| node_llavaarch
node_llavaarch -->|"builds encoders"| node_encoders
node_llavaarch -->|"builds projector"| node_projector
node_projector -->|"selects point projector"| node_pointprojector
node_sgmodel -->|"formats prompts"| node_conversation
node_researcher -->|"inspects take"| node_visualize
node_visualize -->|"reads data"| node_loader
node_visualize -->|"renders timepoint"| node_timepoint
node_timepoint -->|"reads data"| node_loader

click node_merge "https://github.com/ardamamur/egoexor/blob/main/data/utils/merge_h5.py"
click node_loader "https://github.com/ardamamur/egoexor/blob/main/data/utils/load_h5.py"
click node_formatgen "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py"
click node_converters "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/llava_helpers/scene_graph_converters.py"
click node_templates "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/llava_helpers/scene_graph_templates.py"
click node_dataset "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/or_dataset.py"
click node_datasetutils "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/dataset_utils.py"
click node_train "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/train/train.py"
click node_trainer "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/train/llava_trainer.py"
click node_sgmodel "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py"
click node_transform "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/input_transformation.py"
click node_infer "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/main.py"
click node_builder "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/model/builder.py"
click node_llavaarch "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/model/llava_arch.py"
click node_encoders "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/model/multimodal_encoder/builder.py"
click node_projector "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/model/multimodal_projector/builder.py"
click node_pointprojector "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/model/multimodal_projector/pointtransformerv3.py"
click node_conversation "https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/conversation.py"
click node_visualize "https://github.com/ardamamur/egoexor/blob/main/data/utils/visualize_take.py"
click node_timepoint "https://github.com/ardamamur/egoexor/blob/main/data/utils/visualize_timepoint.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_merge,node_loader,node_formatgen,node_converters,node_templates,node_datasetutils,node_researcher toneBlue
class node_builder,node_llavaarch,node_encoders,node_projector,node_pointprojector,node_conversation,node_hdf5 toneAmber
class node_dataset,node_train,node_trainer,node_sgmodel,node_transform,node_infer toneMint
class node_visualize,node_timepoint toneRose
```

## Components

### Dataset and preparation

- [HDF5 merger](https://github.com/ardamamur/egoexor/blob/main/data/utils/merge_h5.py): `data/utils/merge_h5.py`
- [HDF5 reader](https://github.com/ardamamur/egoexor/blob/main/data/utils/load_h5.py): `data/utils/load_h5.py`
- [Sample formatter](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py): `scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py`
- [Graph converters](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/llava_helpers/scene_graph_converters.py): `scene_graph_generation/scene_graph_prediction/llava_helpers/scene_graph_converters.py`
- [Prompt templates](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/llava_helpers/scene_graph_templates.py): `scene_graph_generation/scene_graph_prediction/llava_helpers/scene_graph_templates.py`
- [Graph label utilities](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/dataset_utils.py): `scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/dataset_utils.py`

### Multimodal model

- [Model loader](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/model/builder.py): `scene_graph_generation/LLaVA/llava/model/builder.py`
- [Multimodal integration](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/model/llava_arch.py): `scene_graph_generation/LLaVA/llava/model/llava_arch.py`
- [Modality encoders](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/model/multimodal_encoder/builder.py): `scene_graph_generation/LLaVA/llava/model/multimodal_encoder/builder.py`
- [Feature projectors](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/model/multimodal_projector/builder.py): `scene_graph_generation/LLaVA/llava/model/multimodal_projector/builder.py`
- [Point-cloud projector](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/model/multimodal_projector/pointtransformerv3.py): `scene_graph_generation/LLaVA/llava/model/multimodal_projector/pointtransformerv3.py`
- [Conversation formatting](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/conversation.py): `scene_graph_generation/LLaVA/llava/conversation.py`

### Training and evaluation

- [OR dataset](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/or_dataset.py): `scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/or_dataset.py`
- [LLaVA trainer entry](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/train/train.py): `scene_graph_generation/LLaVA/llava/train/train.py`
- [Training engine](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/LLaVA/llava/train/llava_trainer.py): `scene_graph_generation/LLaVA/llava/train/llava_trainer.py`
- [Scene graph model](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py): `scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py`
- [Input transformation](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/input_transformation.py): `scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/input_transformation.py`
- [Inference and evaluation](https://github.com/ardamamur/egoexor/blob/main/scene_graph_generation/scene_graph_prediction/main.py): `scene_graph_generation/scene_graph_prediction/main.py`

### Data inspection

- [Take visualization](https://github.com/ardamamur/egoexor/blob/main/data/utils/visualize_take.py): `data/utils/visualize_take.py`
- [Timepoint visualization](https://github.com/ardamamur/egoexor/blob/main/data/utils/visualize_timepoint.py): `data/utils/visualize_timepoint.py`

### Other components

- **Dataset user**
- **HDF5 dataset**

## Connections

- Dataset user → HDF5 merger: merges data (evidence: `README.md`)
- HDF5 merger → HDF5 dataset: writes dataset (evidence: `README.md`)
- Dataset user → Sample formatter: generates samples (evidence: `README.md`)
- Sample formatter → Graph converters: uses converters (evidence: `scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py`)
- Sample formatter → Prompt templates: uses templates (evidence: `scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py`)
- Sample formatter → Graph label utilities: uses labels (evidence: `scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py`)
- Dataset user → LLaVA trainer entry: starts training (evidence: `README.md`)
- LLaVA trainer entry → Training engine: uses trainer (evidence: `scene_graph_generation/LLaVA/llava/train/train.py`)
- LLaVA trainer entry → Graph label utilities: uses labels (evidence: `scene_graph_generation/LLaVA/llava/train/train.py`)
- Dataset user → Inference and evaluation: runs benchmark (evidence: `README.md`)
- Inference and evaluation → OR dataset: loads evaluation data (evidence: `scene_graph_generation/scene_graph_prediction/main.py`)
- Inference and evaluation → Scene graph model: evaluates model (evidence: `scene_graph_generation/scene_graph_prediction/main.py`)
- Scene graph model → Model loader: loads model (evidence: `scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py`)
- Scene graph model → Input transformation: transforms inputs (evidence: `scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py`)
- Scene graph model → OR dataset: uses dataset (evidence: `scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py`)
- Scene graph model → Graph label utilities: uses labels (evidence: `scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py`)
- OR dataset → Graph label utilities: uses utilities (evidence: `scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/or_dataset.py`)
- Model loader → Multimodal integration: builds model (evidence: `scene_graph_generation/LLaVA/llava/model/builder.py`)
- Multimodal integration → Modality encoders: builds encoders (evidence: `scene_graph_generation/LLaVA/llava/model/llava_arch.py`)
- Multimodal integration → Feature projectors: builds projector (evidence: `scene_graph_generation/LLaVA/llava/model/llava_arch.py`)
- Feature projectors → Point-cloud projector: selects point projector (evidence: `scene_graph_generation/LLaVA/llava/model/multimodal_projector/builder.py`)
- Scene graph model → Conversation formatting: formats prompts (evidence: `scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py`)
- Dataset user → Take visualization: inspects take (evidence: `README.md`)
- Take visualization → HDF5 reader: reads data (evidence: `data/utils/visualize_take.py`)
- Take visualization → Timepoint visualization: renders timepoint (evidence: `data/utils/visualize_take.py`)
- Timepoint visualization → HDF5 reader: reads data (evidence: `data/utils/visualize_timepoint.py`)
