# eliaswohlgemuth/orsu architecture

> Architecture diagram of the GitHub repository eliaswohlgemuth/orsu, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eliaswohlgemuth/orsu
- Repository: https://github.com/eliaswohlgemuth/orsu

## Overview

##### Main Components
- Query / Input Interface: Show how the system begins with user queries that can either be provided via the demo notebook (demo.ipynb) or through a benchmarking script (main_batch.py). This component is pivotal because its query drives the rest of the architecture.
- Data Ingestion and Benchmark/Dataset Management: Illustrate how multimodal data is handled. Show the dataset part (e.g., OR-multimodal within the data/ folder) where different modalities (images, audio files, 3D point clouds) are stored as well as datasets for benchmarking (queries, CSV files, etc.).
- Configuration and Knowledge Base: Include a separate component for configuration files (configs/ folder) and the knowledge components (contained in GLIP/knowledge) that guide the LLM’s code generation process, including chain-of-thought details and knowledge prompts.

##### Relationships and Interactions
- The Query/Input Interface sends a user query into the LLM Code Generator.
- The LLM component (through chain-of-thought and knowledge integration mechanisms) generates procedural code that defines how to invoke the proper data processing tools.
- The generated code is handed off by the Execution Engine to the appropriate tool interfaces such as GLIP (located within the GLIP directory and further supported by maskrcnn_benchmark) for visual processing, or custom routines (e.g., in utils or scripts like video_segment.py) for audio and other data modalities.
- All processing tools then interact with the underlying multimodal dataset (data/ folder) to perform tasks like object detection, relation detection, and scene graph generation.
- Finally, the processed results are fed back (maybe visualized in demo.ipynb) as outputs, with evaluation happening per benchmarks defined in the configs/benchmarks folder.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Query / Input Interface
    subgraph "Query / Input Interface"
        Q1["Demo Notebook"] 
        Q2["Benchmark Script"]
    end

    %% Configuration and Knowledge Base
    subgraph "Configuration & Knowledge Base"
        CLLM["LLM Config (or-multimodal.yaml)"]
        CKb["Knowledge Base (GLIP/knowledge)"]
        CP["Prompt Files (prompts)"]
    end

    %% LLM Code Generation
    LLM["LLM Code Generator"]

    %% Tool Interface & Execution Engine
    subgraph "Tool Interface & Execution Engine"
        T1["Visual Tool (GLIP)"]
        T2["Detection Modules (maskrcnn_benchmark)"]
        T3["Tool Scripts (tools)"]
        T4["Utility Scripts (utils)"]
    end

    %% Data Ingestion and Benchmark/Dataset Management
    subgraph "Data & Benchmark Management"
        D1["Data Repository (OR-multimodal)"]
        D2["Benchmark Configs (configs/benchmarks)"]
    end

    %% Connections and Data Flow
    Q1 -->|"sends query"| LLM
    Q2 -->|"sends query"| LLM
    CLLM -->|"configures"| LLM
    CKb  -->|"provides knowledge"| LLM
    CP   -->|"supplies prompts"| LLM
    LLM -->|"generates code"| T1
    LLM --> T2
    LLM --> T3
    LLM --> T4
    T1 -->|"processes data"| D1
    T2 --> D1
    T3 --> D1
    T4 --> D1
    D1 -->|"provides benchmark data"| D2
    D2 -->|"evaluation results"| Q1

    %% Click Events
    click Q1 "https://github.com/eliaswohlgemuth/orsu/blob/main/demo.ipynb"
    click Q2 "https://github.com/eliaswohlgemuth/orsu/blob/main/main_batch.py"
    click CLLM "https://github.com/eliaswohlgemuth/orsu/blob/main/configs/or-multimodal.yaml"
    click CKb "https://github.com/eliaswohlgemuth/orsu/tree/main/GLIP/knowledge"
    click CP "https://github.com/eliaswohlgemuth/orsu/tree/main/prompts"
    click T1 "https://github.com/eliaswohlgemuth/orsu/tree/main/GLIP"
    click T2 "https://github.com/eliaswohlgemuth/orsu/tree/main/GLIP/maskrcnn_benchmark"
    click T3 "https://github.com/eliaswohlgemuth/orsu/tree/main/tools"
    click T4 "https://github.com/eliaswohlgemuth/orsu/tree/main/utils"
    click D1 "https://github.com/eliaswohlgemuth/orsu/tree/main/data/OR-multimodal"
    click D2 "https://github.com/eliaswohlgemuth/orsu/tree/main/configs/benchmarks"

    %% Styles
    class Q1,Q2 fill:#66ccff,stroke:#333,stroke-width:2px;
    class CLLM,CKb,CP fill:#ccffcc,stroke:#333,stroke-width:2px;
    class LLM fill:#ffcc00,stroke:#333,stroke-width:2px;
    class T1,T2,T3,T4 fill:#ff9999,stroke:#333,stroke-width:2px;
    class D1,D2 fill:#ccccff,stroke:#333,stroke-width:2px;
```
