How shauli-ravfogel/lm-counterfactuals works
Project type and purpose
• This is an open-source research tool (Python package + scripts + notebooks) for generating “true” counterfactual strings from an input string using pre-trained LMs via the Gumbel-Max trick. It is neither a web app nor a full-stack product, but rather a command-line + notebook pipeline for data → model → counterfactual generation → analysis.
File structure
• Top-level scripts:
• run.py, run_mimic.py – main entry points to generate counterfactuals on different datasets
• .py, analyze_edit.py – compute statistics or evaluate generated counterfactuals
• sampling.py – encapsulates the Gumbel sampling / noise-injection logic
• mimic.py – implements the intervention routines (“hindsight sampling”)
• utils.py – helper routines (I/O, logging, configuration)
• Data directory:
• counterfactuals/ – precomputed .pkl outputs for various models/datasets
• Notebooks:
• example.ipynb, memit-analysis.ipynb, print.ipynb – demonstrations and exploratory analysis
• Configuration:
• requirements.txt – dependencies
• README.md – project overview, setup, usage
Key insights from the README
• Core workflow:
a. Load a dataset (Wikipedia or Bios)
b. Load a language model (via HuggingFace)
c. Forward pass: separate deterministic logits computation from sampling noise
d. Gumbel noise vector responsible for the observed output
e. Reuse that noise under an intervention to generate a counterfactual string
f. Save results to counterfactuals/*.pkl
• Two CLI scripts for different intervention experiments
Main components
a. Data Sources
• Wikipedia Dataset
• Bios Dataset
b. Runner / CLI Layer
• run.py
• run_mimic.py
c. Core Library
• sampling.py (Gumbel-Max trick)
• mimic.py (intervention logic)
• utils.py (I/O, formatting, caching)
d. Model Layer
• Pretrained LMs (GPT-2, Llama variants) loaded via HuggingFace
e. Output Storage
• counterfactuals/*.pkl files
f. Analysis Layer
• .py / analyze_edit.py
• Notebooks (example.ipynb, memit-analysis.ipynb, print.ipynb)
g. Environment & Config
• requirements.txt
• virtual environment
Relationships and interactions
• Data Sources → Runner scripts
• Runner scripts call utils.load_dataset() → returns input strings
• Runner → sampling.generate_gumbel_noise() → calls model.forward()
• generate_gumbel_noise() + mimic.intervene() → produce counterfactual outputs
• Counterfactual outputs → utils.save_pickle() → .pkl in counterfactuals/
• Analysis scripts / Notebooks → utils.load_pickle() → compute metrics / visualizations
Architectural patterns
• Pipeline architecture: clear separation of data ingestion → transformation (sampling + intervention) → storage → analysis
• Modular library: core logic separated into sampling.py, mimic.py, utils.py for reusability
• CLI layer orchestrates the pipeline
Technologies & frameworks
• Python 3
• PyTorch + HuggingFace Transformers
• Gumbel-Max trick implementation in sampling.py
• Notebook ecosystem (Jupyter) for analysis
• Clear component labels (e.g., “Gumbel Sampler (sampling.py)”)