How a-n-u-p-a-m/ire_project works
This repository is a research framework for measuring counterfactual effects in cascaded information retrieval and testing budget-aware repairs. The workflow builds and freezes a retrieval universe, prepares relevance and Stage-1/Stage-2 scores, constructs counterfactual ranking worlds and budget allocations, computes document-level effects, then analyzes query diagnoses and predictor fidelity. Shared modules implement the experiment’s data contracts, ranking semantics, and metrics; separate experiment directories provide real-data and synthetic runners. Several experiment paths are documented or indexed but not fully sampled, so their internal steps are omitted.
Main entry points: engine.py Effect engine · worlds.py Ranking worlds · build_universe.py Real data prep · effects.py Document effects
Components 29
Data and design
- Design config
shared/config.py - Table contracts
shared/schema.py - Relevance labels
shared/qrels.py - Frozen universe
shared/universe.py - Run provenance
shared/provenance.py
Counterfactual core
- Budget allocation
shared/allocation.py - Ranking worlds
shared/worlds.py - Ranking order
shared/ordering.py - Rank utility
shared/utility.py - Document effects
shared/effects.py - Effect strata
shared/strata.py
Experiment runners
- Stage-1 retrieval
experiments_anupam/retrievers.py - Real data prep
experiments_anupam/build_universe.py - Stage-1 runs
experiments_anupam/stage1_over_universe.py - Effect engine
experiments_shubham/engine.py - Synthetic data
shared/synthetic.py - Real-data runner
experiments_shubham/real.py
Analysis and prediction
- Query diagnosis
shared/diagnosis.py - H7 power
experiments_shubham/power_h7.py - Effect analysis
experiments_shubham/analysis.py - Predictor features
experiments_chaitanya/features.py - Effect predictors
experiments_chaitanya/predictors.py - Fidelity metrics
shared/metrics.py - Predictor baselines
shared/baselines.py
Stage-2 scoring
- Stage-2 cache
shared/cache.py - Passage fetch
experiments_chaitanya/passage_fetcher.py - Cross-encoder
experiments_chaitanya/reranker.py - Cache filling
experiments_chaitanya/cache_filler.py
Other components
- Researcher
Connections 31
- Researcher → Real data prep: prepares data
- Real data prep → Frozen universe: builds and freezes
experiments_anupam/build_universe.py - Real data prep → Relevance labels: loads labels
experiments_anupam/build_universe.py - Stage-1 runs → Stage-1 retrieval: runs retrieval
experiments_anupam/stage1_over_universe.py - Stage-1 runs → Frozen universe: uses universe
experiments_anupam/stage1_over_universe.py - Cache filling → Passage fetch: fetches passages
experiments_chaitanya/cache_filler.py - Cache filling → Cross-encoder: scores passages
experiments_chaitanya/cache_filler.py - Cache filling → Stage-2 cache: writes scores
experiments_chaitanya/cache_filler.py - Effect engine → Ranking worlds: builds worlds
experiments_shubham/engine.py - Effect engine → Document effects: computes effects
experiments_shubham/engine.py - Effect engine → Synthetic data: runs synthetic
experiments_shubham/engine.py - Effect engine → Budget allocation: runs budget cases
experiments_shubham/engine.py - Ranking worlds → Budget allocation: allocates candidates
shared/worlds.py - Ranking worlds → Ranking order: orders rankings
shared/worlds.py - Document effects → Rank utility: scores ranks
shared/effects.py - Document effects → Ranking worlds: compares worlds
shared/effects.py - Real-data runner → Effect engine: runs real effects
experiments_shubham/real.py - Real-data runner → Stage-2 cache: reads scores
experiments_shubham/real.py - Effect analysis → Query diagnosis: diagnoses queries
experiments_shubham/analysis.py - Effect analysis → H7 power: tests H7
experiments_shubham/analysis.py - Query diagnosis → Ranking worlds: uses reference worlds
shared/diagnosis.py - H7 power → Query diagnosis: sizes H7 sample
experiments_shubham/power_h7.py - Predictor features → Stage-2 cache: reads scores
experiments_chaitanya/features.py - Effect predictors → Predictor baselines: compares baselines
experiments_chaitanya/predictors.py - Effect predictors → Fidelity metrics: evaluates fidelity
experiments_chaitanya/predictors.py - Fidelity metrics → Effect strata: evaluates by stratum
shared/metrics.py - Design config → Table contracts: configures tables
- Effect engine → Design config: loads design
experiments_shubham/engine.py - Effect analysis → Document effects: reads effect tables
experiments_shubham/analysis.py - Effect predictors → Predictor features: uses features
experiments_chaitanya/run_real_predictors.py - Real data prep → Run provenance: records design
experiments_anupam/build_universe.py