How eliaswohlgemuth/orsu works
Main Components
• Query / Input Interface: Show how the system begins with user queries that can either be provided via the demo notebook (demo.ipynb) or through a benchmarking script (main_batch.py). This component is pivotal because its query drives the rest of the architecture.
• Data Ingestion and Benchmark/Dataset Management: Illustrate how multimodal data is handled. Show the dataset part (e.g., OR-multimodal within the data/ folder) where different modalities (images, audio files, 3D point clouds) are stored as well as datasets for benchmarking (queries, CSV files, etc.).
• Configuration and Knowledge Base: Include a separate component for configuration files (configs/ folder) and the knowledge components (contained in GLIP/knowledge) that guide the LLM’s code generation process, including chain-of-thought details and knowledge prompts.
Relationships and Interactions
• The Query/Input Interface sends a user query into the LLM Code Generator.
• The LLM component (through chain-of-thought and knowledge integration mechanisms) generates procedural code that defines how to invoke the proper data processing tools.
• The generated code is handed off by the Execution Engine to the appropriate tool interfaces such as GLIP (located within the GLIP directory and further supported by maskrcnn_benchmark) for visual processing, or custom routines (e.g., in utils or scripts like video_segment.py) for audio and other data modalities.
• All processing tools then interact with the underlying multimodal dataset (data/ folder) to perform tasks like object detection, relation detection, and scene graph generation.
• Finally, the processed results are fed back (maybe visualized in demo.ipynb) as outputs, with evaluation happening per benchmarks defined in the configs/benchmarks folder.