How andyrdt/refusal_direction works
This repository reproduces research on refusal directions in language models. The main workflow loads harmful and harmless prompt data and a selected model, extracts candidate directions, selects one using refusal, steering, and KL metrics, then evaluates completions and cross-entropy loss with and without interventions. Model-family adapters and activation hooks provide the internal layer for running the same experiments across architectures. The README documents the stages; sampled source confirms dataset loading, orchestration helpers, evaluation logic, and hook behavior, but not every call site.
Main entry points: run_pipeline.py Pipeline Runner · runs Experiment Artifacts · select_direction.py Direction Selection · load_dataset.py Dataset Loader
Components 18
Prompt Data
- Dataset Loader
dataset/load_dataset.py - Prompt Datasets
dataset/splits - Evaluation Sets
dataset/processed
Experiment Workflow
- Pipeline Runner
pipeline/run_pipeline.py - Run Configuration
pipeline/config.py - Candidate Directions
pipeline/submodules/generate_directions.py - Direction Selection
pipeline/submodules/select_direction.py - Selected Direction
pipeline/runs
Model Interventions
- Model Factory
pipeline/model_utils/model_factory.py - Model Adapters
pipeline/model_utils - Activation Hooks
pipeline/utils/hook_utils.py
Evaluation
- Completion Generation
pipeline/run_pipeline.py - Safety Evaluation
pipeline/submodules/evaluate_jailbreak.py - Loss Evaluation
pipeline/submodules/evaluate_loss.py - Experiment Artifacts
pipeline/runs
Other components
- Researcher
- Hugging Face Models
- Together AI Judge
Connections 27
- Researcher → Pipeline Runner: starts run
- Pipeline Runner → Run Configuration: configures
- Pipeline Runner → Dataset Loader: loads prompts
- Dataset Loader → Prompt Datasets: reads splits
- Dataset Loader → Evaluation Sets: reads datasets
- Pipeline Runner → Model Factory: creates model
- Model Factory → Model Adapters: selects adapter
- Model Adapters → Hugging Face Models: loads weights
- Pipeline Runner → Candidate Directions: extracts candidates
- Candidate Directions → Model Adapters: measures activations
- Pipeline Runner → Direction Selection: selects direction
- Direction Selection → Activation Hooks: tests interventions
- Direction Selection → Selected Direction: saves direction
- Pipeline Runner → Completion Generation: generates outputs
- Completion Generation → Model Adapters: generates completions
- Completion Generation → Activation Hooks: applies hooks
- Completion Generation → Dataset Loader: loads evaluation set
- Pipeline Runner → Safety Evaluation: evaluates completions
- Safety Evaluation → Together AI Judge: requests classifications
- Pipeline Runner → Loss Evaluation: evaluates loss
- Loss Evaluation → Activation Hooks: applies hooks
- Pipeline Runner → Experiment Artifacts: writes results
- Candidate Directions → Experiment Artifacts: writes candidates
- Direction Selection → Experiment Artifacts: writes metrics
- Completion Generation → Experiment Artifacts: writes completions
- Safety Evaluation → Experiment Artifacts: writes evaluations
- Loss Evaluation → Experiment Artifacts: writes loss metrics