How stogiannidis/srbench works
Project type and purpose
• SRBench is an open‐source research tool for generating synthetic multimodal data and evaluating various VLMs (e.g., InternVL, OpenAI Vision, MiniCPM-V) on reasoning tasks.
• It contains data‐creation pipelines, model‐interaction utilities, and evaluation scripts.
File structure
Top‐level directories/files:
• scripts/run.sh – main bash entry point
• src/data_creation/ – modules to create synthetic data, images, prompts
• src/utils/ – helper functions for model APIs
• src/eval_*.py and src/inference.py – evaluation and inference orchestrators
Observations:
• No web frontend or database—purely a CLI research pipeline.
• Uses external services: Hugging Face dataset hosting, DALL·E image API.
README for additional insights
• Installation via requirements.txt; data fetched from Hugging Face.
• Usage: bash scripts/run.sh drives the end‐to‐end workflow (data download → data creation → evaluation).
a. Main components
1. Entry Point (scripts/run.sh)
2. Data Storage
• Raw dataset (fetched from Hugging Face)
• Synthetic data outputs (JSON prompts, generated images)
3. Data Creation Module (src/data_creation)
• create_prompts.py
• create_images.py (calls images_dalle.py)
• task‐specific scripts (mrt_easy, mrt_complex, paper_folding)
4. Utilities (src/utils/vlm_helpers.py)
• Wraps APIs for calling different VLM backends
5. Inference/Evaluation Module
• inference.py (common orchestration)
• eval_openai.py / eval_intern.py / eval_mini.py
6. Results Output
• Metrics files, logs, visualization reports (if any)
b. Relationships and data flow
• scripts/run.sh → triggers Data Download (Hugging Face CLI)
• Data Download → Raw Dataset Storage
• Entry Point → Data Creation Module → Synthetic Prompt & Image Storage
• Evaluation Module ← Utilities (vlm_helpers) → VLM Service APIs (OpenAI, InternVL, etc.)
• Evaluation Module → Results Outputs
c. Architectural patterns
• Layered pipeline: Data Ingestion → Data Generation → Model Invocation → Evaluation
• Modular separation: each task script is independent; utilities abstract away API differences.
• Single responsibility: each eval_*.py script focuses on one VLM.
d. Relevant technologies/frameworks
• Python 3.12
• Hugging Face CLI & datasets
• External VLM APIs (OpenAI, InternVL, MiniCPM-V)
• DALL·E image generation API