How wollschlager/geometry-of-refusal works
This research codebase computes and evaluates refusal-related directions and subspaces in language-model activations. The main workflow loads harmful/harmless instruction splits, constructs a supported model wrapper, derives and selects DIM directions, then applies interventions and evaluates generated responses. A separate RDO workflow trains directions, orthogonal or DIM-independent variants, and refusal cones; evaluation supports saved runs and cone samples. The graph includes documented capabilities whose wiring is not fully visible in the excerpts; unsampled implementation details are not expanded.
Main entry points: run_pipeline.py DIM runner · model_factory.py Model factory · run_rdo_pipeline.py RDO evaluator · rdo.py RDO training
Components 24
Datasets
- SaladBench splits
data/saladbench_splits - Dataset loader
refusal_direction/dataset/load_dataset.py - Dataset preparation
data/generate_datasets.py
DIM Pipeline
- DIM runner
refusal_direction/pipeline/run_pipeline.py - Generate directions
refusal_direction/pipeline/submodules/generate_directions.py - Select direction
refusal_direction/pipeline/submodules/select_direction.py
RDO Experiments
- RDO training
rdo.py - RDO evaluator
refusal_direction/pipeline/run_rdo_pipeline.py - Cone samples
refusal_direction/pipeline/run_rdo_samples.py
Model Runtime
- Model factory
refusal_direction/pipeline/model_utils/model_factory.py - Model interface
refusal_direction/pipeline/model_utils/model_base.py - Gemma adapter
refusal_direction/pipeline/model_utils/gemma_model.py - Llama adapter
refusal_direction/pipeline/model_utils/llama3_model.py - Qwen adapter
refusal_direction/pipeline/model_utils/qwen_model.py - Activation hooks
refusal_direction/pipeline/utils/hook_utils.py
Evaluation and Analysis
- Jailbreak evaluation
refusal_direction/pipeline/submodules/evaluate_jailbreak.py - Loss evaluation
refusal_direction/pipeline/submodules/evaluate_loss.py - Scoring utilities
scoring.py - Cosine analysis
cosinesim_analysis.py - Experiment plots
plots.py
Other components
- Researcher
- Weights & Biases
- Hugging Face models
- StrongREJECT
Connections 32
- Researcher → DIM runner: runs DIM
- Researcher → RDO training: trains variants
- Researcher → RDO evaluator: evaluates run
- Researcher → Cone samples: evaluates samples
- Dataset preparation → SaladBench splits: prepares data
- DIM runner → Dataset loader: loads splits
- DIM runner → Model factory: constructs model
- DIM runner → Generate directions: generates candidates
- DIM runner → Select direction: selects candidate
- DIM runner → Jailbreak evaluation: evaluates responses
- DIM runner → Loss evaluation: evaluates loss
- RDO training → Dataset loader: loads data
- RDO training → Model factory: loads model
- RDO training → Weights & Biases: logs experiment
- RDO evaluator → Model factory: constructs model
- RDO evaluator → Select direction: selects direction
- RDO evaluator → Jailbreak evaluation: scores completions
- RDO evaluator → Weights & Biases: retrieves run
- Cone samples → Weights & Biases: retrieves run
- Model factory → Model interface: creates wrapper
- Model factory → Gemma adapter: selects adapter
- Model factory → Llama adapter: selects adapter
- Model factory → Qwen adapter: selects adapter
- Gemma adapter → Hugging Face models: loads weights
- Llama adapter → Hugging Face models: loads weights
- Qwen adapter → Hugging Face models: loads weights
- Select direction → Activation hooks: tests interventions
- RDO evaluator → Activation hooks: applies interventions
- Jailbreak evaluation → StrongREJECT: scores responses
- Researcher → Cosine analysis: analyzes vectors
- Researcher → Experiment plots: summarizes results
- Experiment plots → Weights & Biases: queries runs