How masadi-99/mirage works
MIRAGE is a research framework and benchmark for studying vision-language model behavior when visual inputs are removed. Its main workflow prepares experiment requests from datasets and image-mode settings, runs model inference, then evaluates responses; paper-specific experiments extend this workflow to mirage rates, diagnosis bias, benchmark scores, guess-mode comparisons, super-guesser training, and B-Clean. The README documents these capabilities and the three-stage pipeline; sampled source confirms selected client, evaluator, and experiment behavior, but not every subsystem’s internal wiring.
Main entry points: evaluate_results.py Evaluate results · run_mirage_vs_guess.py Mirage vs guess · loader.py Dataset loading · prepare_experiment.py Prepare requests
Components 24
Core Pipeline
- Prepare requests
scripts/prepare_experiment.py - Run inference
scripts/run_inference.py - Evaluate results
scripts/evaluate_results.py
Core Library
- Dataset loading
mirage/datasets/loader.py - Dataset converters
mirage/datasets/converters - Dataset base
mirage/datasets/base.py - Experiment config
mirage/experiments/config.py - Message formatting
mirage/experiments/message_formatter.py - Prompt templates
mirage/prompts/templates.py - Model registry
mirage/models/base.py - Model clients
mirage/models/clients - Answer comparators
mirage/evaluators/comparators - LLM judge
mirage/evaluators/llm_judge.py - Measurements
mirage/evaluators/measurements.py
Paper Experiments
- Mirage rate
experiments/exp1_mirage_rate - Diagnosis bias
experiments/exp2_mirage_bias - Benchmark evaluation
experiments/exp3_benchmark_eval - Mirage vs guess
experiments/exp4_mirage_vs_guess/run_mirage_vs_guess.py - Super-guesser
experiments/exp5_super_guesser - B-Clean
experiments/exp6_bclean/run_bclean.py
Analysis
- Mirage scores
analysis/compute_mirage_scores.py - Paper figures
analysis
Other components
- Researcher
- Model providers
Connections 30
- Researcher → Prepare requests: configures
- Prepare requests → Dataset loading: loads datasets
- Prepare requests → Experiment config: sets options
- Prepare requests → Message formatting: formats requests
- Message formatting → Prompt templates: uses templates
- Dataset loading → Dataset base: uses
- Dataset loading → Dataset converters: converts data
- Researcher → Run inference: runs
- Run inference → Model registry: selects client
- Model registry → Model clients: dispatches
- Model clients → Model providers: sends requests
- Researcher → Evaluate results: scores results
- Evaluate results → Answer comparators: compares answers
- Evaluate results → LLM judge: judges answers
- LLM judge → Model registry: requests judgments
- Evaluate results → Measurements: computes measures
- Researcher → Mirage rate: runs study
- Researcher → Diagnosis bias: runs study
- Researcher → Benchmark evaluation: runs study
- Researcher → Mirage vs guess: runs comparison
- Mirage vs guess → Model registry: selects client
- Mirage vs guess → Dataset loading: loads datasets
- Mirage vs guess → Answer comparators: scores answers
- Researcher → Super-guesser: trains and evaluates
- Researcher → B-Clean: cleans benchmarks
- B-Clean → Dataset loading: reads datasets
- Evaluate results → Mirage scores: supplies accuracy
- Mirage scores → Paper figures: feeds visualizations
- Mirage rate → Paper figures: supplies measurements
- Mirage vs guess → Paper figures: supplies comparison