How ardamamur/egoexor works
EgoExOR is a multimodal operating-room dataset and scene-graph benchmark, with a baseline model for surgical activity understanding. The documented path runs from HDF5 data preparation and LLaVA-format sample generation through model training and inference, where predictions support scene-graph evaluation. The map also preserves the dataset’s visualization utilities and the baseline’s multimodal and language-model layers. Some subsystem internals are unsampled; links to them are omitted where the available excerpts, source index, or README do not establish a relationship.
Main entry points: scene_graph_prediction_model.py Scene graph model · generate_dataset_format_for_llava.py Sample formatter · dataset_utils.py Graph label utilities · llava_arch.py Multimodal integration
Components 22
Dataset and preparation
- HDF5 merger
data/utils/merge_h5.py - HDF5 reader
data/utils/load_h5.py - Sample formatter
scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py - Graph converters
scene_graph_generation/scene_graph_prediction/llava_helpers/scene_graph_converters.py - Prompt templates
scene_graph_generation/scene_graph_prediction/llava_helpers/scene_graph_templates.py - Graph label utilities
scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/dataset_utils.py
Multimodal model
- Model loader
scene_graph_generation/LLaVA/llava/model/builder.py - Multimodal integration
scene_graph_generation/LLaVA/llava/model/llava_arch.py - Modality encoders
scene_graph_generation/LLaVA/llava/model/multimodal_encoder/builder.py - Feature projectors
scene_graph_generation/LLaVA/llava/model/multimodal_projector/builder.py - Point-cloud projector
scene_graph_generation/LLaVA/llava/model/multimodal_projector/pointtransformerv3.py - Conversation formatting
scene_graph_generation/LLaVA/llava/conversation.py
Training and evaluation
- OR dataset
scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/or_dataset.py - LLaVA trainer entry
scene_graph_generation/LLaVA/llava/train/train.py - Training engine
scene_graph_generation/LLaVA/llava/train/llava_trainer.py - Scene graph model
scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py - Input transformation
scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/input_transformation.py - Inference and evaluation
scene_graph_generation/scene_graph_prediction/main.py
Data inspection
- Take visualization
data/utils/visualize_take.py - Timepoint visualization
data/utils/visualize_timepoint.py
Other components
- Dataset user
- HDF5 dataset
Connections 26
- Dataset user → HDF5 merger: merges data
README.md - HDF5 merger → HDF5 dataset: writes dataset
README.md - Dataset user → Sample formatter: generates samples
README.md - Sample formatter → Graph converters: uses converters
scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py - Sample formatter → Prompt templates: uses templates
scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py - Sample formatter → Graph label utilities: uses labels
scene_graph_generation/scene_graph_prediction/llava_helpers/generate_dataset_format_for_llava.py - Dataset user → LLaVA trainer entry: starts training
README.md - LLaVA trainer entry → Training engine: uses trainer
scene_graph_generation/LLaVA/llava/train/train.py - LLaVA trainer entry → Graph label utilities: uses labels
scene_graph_generation/LLaVA/llava/train/train.py - Dataset user → Inference and evaluation: runs benchmark
README.md - Inference and evaluation → OR dataset: loads evaluation data
scene_graph_generation/scene_graph_prediction/main.py - Inference and evaluation → Scene graph model: evaluates model
scene_graph_generation/scene_graph_prediction/main.py - Scene graph model → Model loader: loads model
scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py - Scene graph model → Input transformation: transforms inputs
scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py - Scene graph model → OR dataset: uses dataset
scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py - Scene graph model → Graph label utilities: uses labels
scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py - OR dataset → Graph label utilities: uses utilities
scene_graph_generation/scene_graph_prediction/scene_graph_helpers/dataset/or_dataset.py - Model loader → Multimodal integration: builds model
scene_graph_generation/LLaVA/llava/model/builder.py - Multimodal integration → Modality encoders: builds encoders
scene_graph_generation/LLaVA/llava/model/llava_arch.py - Multimodal integration → Feature projectors: builds projector
scene_graph_generation/LLaVA/llava/model/llava_arch.py - Feature projectors → Point-cloud projector: selects point projector
scene_graph_generation/LLaVA/llava/model/multimodal_projector/builder.py - Scene graph model → Conversation formatting: formats prompts
scene_graph_generation/scene_graph_prediction/scene_graph_helpers/model/scene_graph_prediction_model.py - Dataset user → Take visualization: inspects take
README.md - Take visualization → HDF5 reader: reads data
data/utils/visualize_take.py - Take visualization → Timepoint visualization: renders timepoint
data/utils/visualize_take.py - Timepoint visualization → HDF5 reader: reads data
data/utils/visualize_timepoint.py