# stogiannidis/srbench architecture

> Architecture diagram of the GitHub repository stogiannidis/srbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stogiannidis/srbench
- Repository: https://github.com/stogiannidis/srbench

## Overview

##### Project type and purpose
- SRBench is an open‐source research tool for generating synthetic multimodal data and evaluating various VLMs (e.g., InternVL, OpenAI Vision, MiniCPM-V) on reasoning tasks.
- It contains data‐creation pipelines, model‐interaction utilities, and evaluation scripts.

##### File structure
Top‐level directories/files:
- scripts/run.sh – main bash entry point
- src/data_creation/ – modules to create synthetic data, images, prompts
- src/utils/ – helper functions for model APIs
- src/eval_*.py and src/inference.py – evaluation and inference orchestrators

Observations:
- No web frontend or database—purely a CLI research pipeline.
- Uses external services: Hugging Face dataset hosting, DALL·E image API.

##### README for additional insights
- Installation via requirements.txt; data fetched from Hugging Face.
- Usage: `bash scripts/run.sh` drives the end‐to‐end workflow (data download → data creation → evaluation).

a. Main components
1. Entry Point (scripts/run.sh)
2. Data Storage
- Raw dataset (fetched from Hugging Face)
- Synthetic data outputs (JSON prompts, generated images)
3. Data Creation Module (src/data_creation)
- create_prompts.py
- create_images.py (calls images_dalle.py)
- task‐specific scripts (mrt_easy, mrt_complex, paper_folding)
4. Utilities (src/utils/vlm_helpers.py)
- Wraps APIs for calling different VLM backends
5. Inference/Evaluation Module
- inference.py (common orchestration)
- eval_openai.py / eval_intern.py / eval_mini.py
6. Results Output
- Metrics files, logs, visualization reports (if any)

b. Relationships and data flow
- scripts/run.sh → triggers Data Download (Hugging Face CLI)
- Data Download → Raw Dataset Storage
- Entry Point → Data Creation Module → Synthetic Prompt & Image Storage
- Evaluation Module ← Utilities (vlm_helpers) → VLM Service APIs (OpenAI, InternVL, etc.)
- Evaluation Module → Results Outputs

c. Architectural patterns
- Layered pipeline: Data Ingestion → Data Generation → Model Invocation → Evaluation
- Modular separation: each task script is independent; utilities abstract away API differences.
- Single responsibility: each eval_*.py script focuses on one VLM.

d. Relevant technologies/frameworks
- Python 3.12
- Hugging Face CLI & datasets
- External VLM APIs (OpenAI, InternVL, MiniCPM-V)
- DALL·E image generation API

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Entry Point
    Entry["Entry Point\nscripts/run.sh"]:::internal
    click Entry "https://github.com/stogiannidis/srbench/blob/main/scripts/run.sh"

    %% Data Ingestion
    HFCLI["Hugging Face CLI"]:::external
    RawData["Raw Dataset"]:::storage

    Entry -->|"download data"| HFCLI
    HFCLI -->|"stores raw data"| RawData

    %% Data Creation Module
    subgraph "Data Creation Module" 
        direction TB
        CreateData["create_data.py\n(orchestrator)"]:::internal
        click CreateData "https://github.com/stogiannidis/srbench/blob/main/src/data_creation/create_data.py"

        CreatePrompts["create_prompts.py"]:::internal
        click CreatePrompts "https://github.com/stogiannidis/srbench/blob/main/src/data_creation/create_prompts.py"

        CreateImages["create_images.py"]:::internal
        click CreateImages "https://github.com/stogiannidis/srbench/blob/main/src/data_creation/create_images.py"

        ImagesDalle["images_dalle.py\n(DALL·E wrapper)"]:::internal
        click ImagesDalle "https://github.com/stogiannidis/srbench/blob/main/src/data_creation/images_dalle.py"

        subgraph "Task-Specific Scripts"
            direction TB
            MRT_Easy["mrt_easy.py"]:::internal
            click MRT_Easy "https://github.com/stogiannidis/srbench/blob/main/src/data_creation/mrt_easy.py"
            MRT_Complex["mrt_complex.py"]:::internal
            click MRT_Complex "https://github.com/stogiannidis/srbench/blob/main/src/data_creation/mrt_complex.py"
            PaperFold["paper_folding.py"]:::internal
            click PaperFold "https://github.com/stogiannidis/srbench/blob/main/src/data_creation/paper_folding.py"
        end
    end

    RawData -->|"input to"| CreateData
    CreateData --> CreatePrompts
    CreatePrompts -->|"prompts output"| SyntheticData["Synthetic Data Store"]:::storage

    CreateData --> CreateImages
    CreateImages --> ImagesDalle
    ImagesDalle -->|"images output"| SyntheticData

    CreateData --> MRT_Easy
    CreateData --> MRT_Complex
    CreateData --> PaperFold
    MRT_Easy --> SyntheticData
    MRT_Complex --> SyntheticData
    PaperFold --> SyntheticData

    %% Utilities
    Utils["vlm_helpers.py"]:::internal
    click Utils "https://github.com/stogiannidis/srbench/blob/main/src/utils/vlm_helpers.py"

    %% Inference and Evaluation
    Inference["inference.py\n(orchestrator)"]:::internal
    click Inference "https://github.com/stogiannidis/srbench/blob/main/src/inference.py"

    subgraph "Evaluation Modules"
        direction TB
        EvalOpenAI["eval_openai.py"]:::internal
        click EvalOpenAI "https://github.com/stogiannidis/srbench/blob/main/src/eval_openai.py"
        EvalIntern["eval_intern.py"]:::internal
        click EvalIntern "https://github.com/stogiannidis/srbench/blob/main/src/eval_intern.py"
        EvalMini["eval_mini.py"]:::internal
        click EvalMini "https://github.com/stogiannidis/srbench/blob/main/src/eval_mini.py"
    end

    SyntheticData --> Inference
    Utils --> Inference
    Inference --> EvalOpenAI
    Inference --> EvalIntern
    Inference --> EvalMini
    Utils --> EvalOpenAI
    Utils --> EvalIntern
    Utils --> EvalMini

    %% External Services
    subgraph "External Services"
        direction TB
        OpenAIAPI["OpenAI Vision API"]:::external
        InternVLAPI["InternVL API"]:::external
        MiniCPMAPI["MiniCPM-V API"]:::external
        DALLEAPI["DALL·E API"]:::external
    end

    EvalOpenAI -->|"calls API"| OpenAIAPI
    EvalIntern -->|"calls API"| InternVLAPI
    EvalMini -->|"calls API"| MiniCPMAPI
    CreateImages -->|"image gen"| DALLEAPI

    %% Results Output
    Results["Results Output\n(metrics, logs)"]:::storage

    EvalOpenAI -->|"writes metrics"| Results
    EvalIntern -->|"writes metrics"| Results
    EvalMini -->|"writes metrics"| Results

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#000,stroke-width:1px
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px
```
