How pranavagrawai/raa works
Project type and purpose
• RaA Benchmark is a Python-based CLI pipeline that runs a configurable, automated “telephone” game between modalities (image↔text), measures semantic drift after each iteration, and produces evaluation metrics plus visual and narrative reports.
File structure
• Top-level directories:
• configs: YAML files defining experiment parameters
• data: seed inputs (images, text) for experiments
• src: Python modules implementing core logic
• results: structured output artifacts (generated images/text, ratings, charts, summaries)
• tests: unit tests for each component
• Key modules in src:
• main.py – orchestrates end-to-end pipeline
• benchmark_config.py – loads & validates YAML config
• loop_controller.py – drives the I-T-I or T-I-T iterations
• prompt_engine.py – calls external generative model APIs
• evaluation_engine.py – runs pairwise comparisons via a multimodal LLM
• graph_creator.py – plots quantitative charts
• reporting_summary.py – composes qualitative drift summaries
• output_manager.py – handles file I/O into results/
README for additional insights
• The workflow is strictly linear/pipelined: configuration → seed data ingestion → iterative generation → evaluation → reporting → output.
• External dependency: AI model API (e.g., Google Gemini) accessed via prompt_engine using credentials from .env.
Essential architectural elements
• That all file reads/writes are managed by Output Manager into the results directory
• Tests suite as a side-band quality check