# masadi-99/mirage architecture

> Architecture diagram of the GitHub repository masadi-99/mirage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/masadi-99/mirage
- Repository: https://github.com/masadi-99/mirage

## Overview

MIRAGE is a research framework and benchmark for studying vision-language model behavior when visual inputs are removed. Its main workflow prepares experiment requests from datasets and image-mode settings, runs model inference, then evaluates responses; paper-specific experiments extend this workflow to mirage rates, diagnosis bias, benchmark scores, guess-mode comparisons, super-guesser training, and B-Clean. The README documents these capabilities and the three-stage pipeline; sampled source confirms selected client, evaluator, and experiment behavior, but not every subsystem’s internal wiring.

Main entry points: [`scripts/evaluate_results.py`](https://github.com/masadi-99/mirage/blob/main/scripts/evaluate_results.py) (Evaluate results), [`experiments/exp4_mirage_vs_guess/run_mirage_vs_guess.py`](https://github.com/masadi-99/mirage/blob/main/experiments/exp4_mirage_vs_guess/run_mirage_vs_guess.py) (Mirage vs guess), [`mirage/datasets/loader.py`](https://github.com/masadi-99/mirage/blob/main/mirage/datasets/loader.py) (Dataset loading), [`scripts/prepare_experiment.py`](https://github.com/masadi-99/mirage/blob/main/scripts/prepare_experiment.py) (Prepare requests)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["Core Pipeline"]
  node_prepare["Prepare requests"]
  node_infer["Run inference<br/>[run_inference.py]"]
  node_evaluate["Evaluate results"]
end

subgraph group_library["Core Library"]
  node_dataset_loader["Dataset loading<br/>[loader.py]"]
  node_converters["Dataset converters"]
  node_dataset_base["Dataset base<br/>[base.py]"]
  node_experiment_config["Experiment config<br/>[config.py]"]
  node_formatter["Message formatting"]
  node_prompts["Prompt templates<br/>[templates.py]"]
  node_model_registry["Model registry<br/>[base.py]"]
  node_clients["Model clients"]
  node_comparators["Answer comparators"]
  node_llm_judge["LLM judge<br/>[llm_judge.py]"]
  node_measurements["Measurements<br/>[measurements.py]"]
end

subgraph group_experiments["Paper Experiments"]
  node_exp1["Mirage rate"]
  node_exp2["Diagnosis bias"]
  node_exp3["Benchmark evaluation"]
  node_exp4["Mirage vs guess"]
  node_exp5["Super-guesser"]
  node_exp6["B-Clean<br/>[run_bclean.py]"]
end

subgraph group_analysis["Analysis"]
  node_score_analysis["Mirage scores"]
  node_figures["Paper figures"]
end

node_researcher(("Researcher"))
node_vlm_services{{"Model providers"}}

node_researcher -->|"configures"| node_prepare
node_prepare -->|"loads datasets"| node_dataset_loader
node_prepare -->|"sets options"| node_experiment_config
node_prepare -->|"formats requests"| node_formatter
node_formatter -->|"uses templates"| node_prompts
node_dataset_loader -->|"uses"| node_dataset_base
node_dataset_loader -->|"converts data"| node_converters
node_researcher -->|"runs"| node_infer
node_infer -->|"selects client"| node_model_registry
node_model_registry -->|"dispatches"| node_clients
node_clients -->|"sends requests"| node_vlm_services
node_researcher -->|"scores results"| node_evaluate
node_evaluate -->|"compares answers"| node_comparators
node_evaluate -->|"judges answers"| node_llm_judge
node_llm_judge -->|"requests judgments"| node_model_registry
node_evaluate -->|"computes measures"| node_measurements
node_researcher -->|"runs study"| node_exp1
node_researcher -->|"runs study"| node_exp2
node_researcher -->|"runs study"| node_exp3
node_researcher -->|"runs comparison"| node_exp4
node_exp4 -->|"selects client"| node_model_registry
node_exp4 -->|"loads datasets"| node_dataset_loader
node_exp4 -->|"scores answers"| node_comparators
node_researcher -->|"trains and evaluates"| node_exp5
node_researcher -->|"cleans benchmarks"| node_exp6
node_exp6 -->|"reads datasets"| node_dataset_loader
node_evaluate -.->|"supplies accuracy"| node_score_analysis
node_score_analysis -.->|"feeds visualizations"| node_figures
node_exp1 -.->|"supplies measurements"| node_figures
node_exp4 -.->|"supplies comparison"| node_figures

click node_prepare "https://github.com/masadi-99/mirage/blob/main/scripts/prepare_experiment.py"
click node_infer "https://github.com/masadi-99/mirage/blob/main/scripts/run_inference.py"
click node_evaluate "https://github.com/masadi-99/mirage/blob/main/scripts/evaluate_results.py"
click node_dataset_loader "https://github.com/masadi-99/mirage/blob/main/mirage/datasets/loader.py"
click node_converters "https://github.com/masadi-99/mirage/tree/main/mirage/datasets/converters"
click node_dataset_base "https://github.com/masadi-99/mirage/blob/main/mirage/datasets/base.py"
click node_experiment_config "https://github.com/masadi-99/mirage/blob/main/mirage/experiments/config.py"
click node_formatter "https://github.com/masadi-99/mirage/blob/main/mirage/experiments/message_formatter.py"
click node_prompts "https://github.com/masadi-99/mirage/blob/main/mirage/prompts/templates.py"
click node_model_registry "https://github.com/masadi-99/mirage/blob/main/mirage/models/base.py"
click node_clients "https://github.com/masadi-99/mirage/tree/main/mirage/models/clients"
click node_comparators "https://github.com/masadi-99/mirage/tree/main/mirage/evaluators/comparators"
click node_llm_judge "https://github.com/masadi-99/mirage/blob/main/mirage/evaluators/llm_judge.py"
click node_measurements "https://github.com/masadi-99/mirage/blob/main/mirage/evaluators/measurements.py"
click node_exp1 "https://github.com/masadi-99/mirage/tree/main/experiments/exp1_mirage_rate"
click node_exp2 "https://github.com/masadi-99/mirage/tree/main/experiments/exp2_mirage_bias"
click node_exp3 "https://github.com/masadi-99/mirage/tree/main/experiments/exp3_benchmark_eval"
click node_exp4 "https://github.com/masadi-99/mirage/blob/main/experiments/exp4_mirage_vs_guess/run_mirage_vs_guess.py"
click node_exp5 "https://github.com/masadi-99/mirage/tree/main/experiments/exp5_super_guesser"
click node_exp6 "https://github.com/masadi-99/mirage/blob/main/experiments/exp6_bclean/run_bclean.py"
click node_score_analysis "https://github.com/masadi-99/mirage/blob/main/analysis/compute_mirage_scores.py"
click node_figures "https://github.com/masadi-99/mirage/tree/main/analysis"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_prepare,node_infer,node_evaluate toneBlue
class node_dataset_loader,node_converters,node_dataset_base,node_experiment_config,node_formatter,node_prompts,node_model_registry,node_clients,node_comparators,node_llm_judge,node_measurements toneAmber
class node_exp1,node_exp2,node_exp3,node_exp4,node_exp5,node_exp6 toneMint
class node_score_analysis,node_figures toneRose
class node_researcher,node_vlm_services toneIndigo
```

## Components

### Core Pipeline

- [Prepare requests](https://github.com/masadi-99/mirage/blob/main/scripts/prepare_experiment.py): `scripts/prepare_experiment.py`
- [Run inference](https://github.com/masadi-99/mirage/blob/main/scripts/run_inference.py): `scripts/run_inference.py`
- [Evaluate results](https://github.com/masadi-99/mirage/blob/main/scripts/evaluate_results.py): `scripts/evaluate_results.py`

### Core Library

- [Dataset loading](https://github.com/masadi-99/mirage/blob/main/mirage/datasets/loader.py): `mirage/datasets/loader.py`
- [Dataset converters](https://github.com/masadi-99/mirage/tree/main/mirage/datasets/converters): `mirage/datasets/converters`
- [Dataset base](https://github.com/masadi-99/mirage/blob/main/mirage/datasets/base.py): `mirage/datasets/base.py`
- [Experiment config](https://github.com/masadi-99/mirage/blob/main/mirage/experiments/config.py): `mirage/experiments/config.py`
- [Message formatting](https://github.com/masadi-99/mirage/blob/main/mirage/experiments/message_formatter.py): `mirage/experiments/message_formatter.py`
- [Prompt templates](https://github.com/masadi-99/mirage/blob/main/mirage/prompts/templates.py): `mirage/prompts/templates.py`
- [Model registry](https://github.com/masadi-99/mirage/blob/main/mirage/models/base.py): `mirage/models/base.py`
- [Model clients](https://github.com/masadi-99/mirage/tree/main/mirage/models/clients): `mirage/models/clients`
- [Answer comparators](https://github.com/masadi-99/mirage/tree/main/mirage/evaluators/comparators): `mirage/evaluators/comparators`
- [LLM judge](https://github.com/masadi-99/mirage/blob/main/mirage/evaluators/llm_judge.py): `mirage/evaluators/llm_judge.py`
- [Measurements](https://github.com/masadi-99/mirage/blob/main/mirage/evaluators/measurements.py): `mirage/evaluators/measurements.py`

### Paper Experiments

- [Mirage rate](https://github.com/masadi-99/mirage/tree/main/experiments/exp1_mirage_rate): `experiments/exp1_mirage_rate`
- [Diagnosis bias](https://github.com/masadi-99/mirage/tree/main/experiments/exp2_mirage_bias): `experiments/exp2_mirage_bias`
- [Benchmark evaluation](https://github.com/masadi-99/mirage/tree/main/experiments/exp3_benchmark_eval): `experiments/exp3_benchmark_eval`
- [Mirage vs guess](https://github.com/masadi-99/mirage/blob/main/experiments/exp4_mirage_vs_guess/run_mirage_vs_guess.py): `experiments/exp4_mirage_vs_guess/run_mirage_vs_guess.py`
- [Super-guesser](https://github.com/masadi-99/mirage/tree/main/experiments/exp5_super_guesser): `experiments/exp5_super_guesser`
- [B-Clean](https://github.com/masadi-99/mirage/blob/main/experiments/exp6_bclean/run_bclean.py): `experiments/exp6_bclean/run_bclean.py`

### Analysis

- [Mirage scores](https://github.com/masadi-99/mirage/blob/main/analysis/compute_mirage_scores.py): `analysis/compute_mirage_scores.py`
- [Paper figures](https://github.com/masadi-99/mirage/tree/main/analysis): `analysis`

### Other components

- **Researcher**
- **Model providers**

## Connections

- Researcher → Prepare requests: configures
- Prepare requests → Dataset loading: loads datasets
- Prepare requests → Experiment config: sets options
- Prepare requests → Message formatting: formats requests
- Message formatting → Prompt templates: uses templates
- Dataset loading → Dataset base: uses
- Dataset loading → Dataset converters: converts data
- Researcher → Run inference: runs
- Run inference → Model registry: selects client
- Model registry → Model clients: dispatches
- Model clients → Model providers: sends requests
- Researcher → Evaluate results: scores results
- Evaluate results → Answer comparators: compares answers
- Evaluate results → LLM judge: judges answers
- LLM judge → Model registry: requests judgments
- Evaluate results → Measurements: computes measures
- Researcher → Mirage rate: runs study
- Researcher → Diagnosis bias: runs study
- Researcher → Benchmark evaluation: runs study
- Researcher → Mirage vs guess: runs comparison
- Mirage vs guess → Model registry: selects client
- Mirage vs guess → Dataset loading: loads datasets
- Mirage vs guess → Answer comparators: scores answers
- Researcher → Super-guesser: trains and evaluates
- Researcher → B-Clean: cleans benchmarks
- B-Clean → Dataset loading: reads datasets
- Evaluate results → Mirage scores: supplies accuracy
- Mirage scores → Paper figures: feeds visualizations
- Mirage rate → Paper figures: supplies measurements
- Mirage vs guess → Paper figures: supplies comparison