How huahenry/branchmi works
BranchMI Pilot is a counterfactual study tool that tests whether token-level BranchMI scores identify reasoning steps that affect a model’s final answer. The run workflow loads configured problems, generates baseline reasoning, selects checkpoints, probes alternative continuations, compares them with oracle outcomes, and analyzes score quality into a report and continue/stop recommendation. The README is TBD, so capability details come from the sampled source and configuration; some experiment internals are only partially excerpted.
Main entry points: experiment.py Pilot experiment · analysis.py Run analysis · cli.py Run CLI · backend.py Transformers backend
Components 19
Experiment run
- Run CLI
branchmi_pilot/src/branchmi_pilot/cli.py - Configuration
branchmi_pilot/src/branchmi_pilot/config.py - Pilot experiment
branchmi_pilot/src/branchmi_pilot/experiment.py - Problem loading
branchmi_pilot/src/branchmi_pilot/data.py - Run records
branchmi_pilot/src/branchmi_pilot/io_utils.py
Model inference
- Prompt construction
branchmi_pilot/src/branchmi_pilot/prompting.py - Transformers backend
branchmi_pilot/src/branchmi_pilot/backend.py - Baseline generation
branchmi_pilot/src/branchmi_pilot/experiment.py - Checkpoint selection
branchmi_pilot/src/branchmi_pilot/scoring.py - Branch probes
branchmi_pilot/src/branchmi_pilot/experiment.py
Answer evaluation
- Oracle comparison
branchmi_pilot/src/branchmi_pilot/experiment.py - Answer evaluation
branchmi_pilot/src/branchmi_pilot/answers.py - BranchMI scoring
branchmi_pilot/src/branchmi_pilot/scoring.py
Study analysis
- Run analysis
branchmi_pilot/src/branchmi_pilot/analysis.py - Metrics and report
branchmi_pilot/src/branchmi_pilot/analysis.py - Analysis I/O
branchmi_pilot/src/branchmi_pilot/io_utils.py
Other components
- Researcher
- Dataset source
- Pretrained model
Connections 25
- Researcher → Run CLI: invokes
- Run CLI → Configuration: loads config
- Run CLI → Pilot experiment: starts run
- Run CLI → Run analysis: requests analysis
- Pilot experiment → Problem loading: loads problems
- Problem loading → Dataset source: loads data
- Pilot experiment → Transformers backend: creates backend
- Transformers backend → Pretrained model: loads model
- Pilot experiment → Prompt construction: builds prompts
- Pilot experiment → Baseline generation: generates baseline
- Baseline generation → Transformers backend: generates tokens
- Pilot experiment → Checkpoint selection: selects positions
- Pilot experiment → Branch probes: runs probes
- Branch probes → Transformers backend: generates branches
- Pilot experiment → Oracle comparison: generates oracle
- Oracle comparison → Transformers backend: generates continuation
- Pilot experiment → Answer evaluation: evaluates answers
- Pilot experiment → BranchMI scoring: computes BranchMI
- Pilot experiment → Run records: writes records
- Run CLI → Pilot experiment: passes config
- Pilot experiment → Run analysis: analyzes run
- Run analysis → Analysis I/O: reads records
- Run analysis → Analysis I/O: writes summary
- Run analysis → BranchMI scoring: computes metrics
- Run analysis → Metrics and report: produces report