How dmbeaglehole/universal_steering works
This is a research/benchmarking codebase for steering LLM/VLM behavior via learned activation directions (latent “control vectors” applied at inference). It has two user-facing workflows:
Steering Benchmark pipeline (batch evaluation across many concepts + models)
• compute directions → generate steered text → judge with GPT-4o → aggregate metrics
VLM Steering (interactive/examples for multimodal steering)
• load a vision-language model → generate with steering on text+image
• offline direction learning from labeled concept data
• online inference-time steering via hooks into model hidden states
• evaluation subsystem using an external judge (OpenAI GPT-4o)
• artifacts (directions, generations, scores/CSVs) flowing through the pipeline
• steering_benchmark/ (pipeline orchestrators)
• run.py — compute steering directions
• eval_generations.py — produce steered outputs
• parse_results.py — evaluate generations using GPT-4o prompts
• read_csv.py — aggregate results
• vlm_steering/ (VLM steering entrypoint)
• main.py — model selection + generation examples
• Core steering library (shared by both workflows)
• neural_controllers.py — “controller” abstraction that applies directions (the central runtime primitive)
• control_toolkits.py — algorithms to compute directions (RFM, linear/logistic, PCA, mean-diff)
• direction_utils.py — hidden-state extraction (forward hooks / layer captures)
• generation_utils.py — generation routines with hooks / controller injection
• rfm.py — RFM algorithm implementation
• utils.py — dataset loaders + general utilities
• Data + prompts
• data/ — concept lists / class files (personalities, moods, fears, places, personas, etc.)
• evaluation_prompts/ — GPT-4o judging templates (multiple versions)
Pipeline architecture / staged batch processing (benchmark)
• each stage is separately runnable, produces artifacts consumed by the next
Controller pattern (neural_controllers.py)
• encapsulates steering behavior independent of model type
Pluggable “direction computation strategies” (control_toolkits.py)
• RFM / linear / logistic / PCA / mean-diff are interchangeable strategies
External evaluation service dependency
• OpenAI GPT-4o is a network boundary + cost/rate-limit boundary
Data-driven concept taxonomy
• concept lists and class definitions live in data/, making benchmark extensible by adding files
Aggregation (steering_benchmark/read_csv.py)
Within stages 1 and 2, show internal calls to:
• utils.py (data load)
• direction_utils.py (activation capture)
• control_toolkits.py + rfm.py (direction learning)
• neural_controllers.py + generation_utils.py (steered decoding)
“Don’t overthink it”: break into many components (recommended component list)
• steering_benchmark.run (Direction Builder)
• control_toolkits (Direction Methods Registry)
• rfm (RFM implementation)
• direction_utils (Activation Capture / Hooks)
• utils (Dataset + misc utilities)
• Output: “Directions store”
• steering_benchmark.eval_generations (Generation Runner)
• generation_utils (Steered decoding / sampling)
• neural_controllers (Controller runtime)
• Output: “Generations store”
• steering_benchmark.parse_results (Judge Runner)
• evaluation_prompts/* (Prompt templates)
• External: “OpenAI GPT-4o”
• Output: “Scores store”
• steering_benchmark.read_csv (Aggregator)
• Output: “Final metrics/report”
• vlm_steering.main (VLM demo runner)
• External: “HF LLMs/VLMs (Llama, Phi, Falcon, Mistral, LLaVA, Gemma…)”
• External: “GPU/CUDA”