# dmbeaglehole/universal_steering architecture

> Architecture diagram of the GitHub repository dmbeaglehole/universal_steering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dmbeaglehole/universal_steering
- Repository: https://github.com/dmbeaglehole/universal_steering

## Overview

This is a **research/benchmarking codebase** for **steering LLM/VLM behavior via learned activation directions** (latent “control vectors” applied at inference). It has **two user-facing workflows**:

##### Steering Benchmark pipeline (batch evaluation across many concepts + models)
- compute directions → generate steered text → judge with GPT-4o → aggregate metrics

##### VLM Steering (interactive/examples for multimodal steering)
- load a vision-language model → generate with steering on text+image

- **offline direction learning** from labeled concept data
- **online inference-time steering** via hooks into model hidden states
- **evaluation subsystem** using an external judge (OpenAI GPT-4o)
- **artifacts** (directions, generations, scores/CSVs) flowing through the pipeline

- **`steering_benchmark/`** (pipeline orchestrators)
- `run.py` — compute steering directions
- `eval_generations.py` — produce steered outputs
- `parse_results.py` — evaluate generations using GPT-4o prompts
- `read_csv.py` — aggregate results

- **`vlm_steering/`** (VLM steering entrypoint)
- `main.py` — model selection + generation examples

- **Core steering library (shared by both workflows)**
- `neural_controllers.py` — “controller” abstraction that applies directions (the central runtime primitive)
- `control_toolkits.py` — algorithms to compute directions (RFM, linear/logistic, PCA, mean-diff)
- `direction_utils.py` — hidden-state extraction (forward hooks / layer captures)
- `generation_utils.py` — generation routines with hooks / controller injection
- `rfm.py` — RFM algorithm implementation
- `utils.py` — dataset loaders + general utilities

- **Data + prompts**
- `data/` — concept lists / class files (personalities, moods, fears, places, personas, etc.)
- `evaluation_prompts/` — GPT-4o judging templates (multiple versions)

##### Pipeline architecture / staged batch processing (benchmark)
- each stage is separately runnable, produces artifacts consumed by the next
##### Controller pattern (`neural_controllers.py`)
- encapsulates steering behavior independent of model type
##### Pluggable “direction computation strategies” (`control_toolkits.py`)
- RFM / linear / logistic / PCA / mean-diff are interchangeable strategies
##### External evaluation service dependency
- OpenAI GPT-4o is a network boundary + cost/rate-limit boundary
##### Data-driven concept taxonomy
- concept lists and class definitions live in `data/`, making benchmark extensible by adding files

##### Aggregation (`steering_benchmark/read_csv.py`)

Within stages 1 and 2, show internal calls to:
- `utils.py` (data load)
- `direction_utils.py` (activation capture)
- `control_toolkits.py` + `rfm.py` (direction learning)
- `neural_controllers.py` + `generation_utils.py` (steered decoding)

##### “Don’t overthink it”: break into many components (recommended component list)
- `steering_benchmark.run` (Direction Builder)
- `control_toolkits` (Direction Methods Registry)
- `rfm` (RFM implementation)
- `direction_utils` (Activation Capture / Hooks)
- `utils` (Dataset + misc utilities)
- Output: “Directions store”
- `steering_benchmark.eval_generations` (Generation Runner)
- `generation_utils` (Steered decoding / sampling)
- `neural_controllers` (Controller runtime)
- Output: “Generations store”
- `steering_benchmark.parse_results` (Judge Runner)
- `evaluation_prompts/*` (Prompt templates)
- External: “OpenAI GPT-4o”
- Output: “Scores store”
- `steering_benchmark.read_csv` (Aggregator)
- Output: “Final metrics/report”
- `vlm_steering.main` (VLM demo runner)
- External: “HF LLMs/VLMs (Llama, Phi, Falcon, Mistral, LLaVA, Gemma…)”
- External: “GPU/CUDA”

## Diagram (Mermaid)

```mermaid
flowchart TD

%% =========================
%% Level 1 — Context Diagram
%% =========================
subgraph "Level 1 — Context (System + External Dependencies)"
direction TB

User["User / CLI / Notebook"]:::external

Repo["Universal Steering Repo (this codebase)"]:::system

HF["Hugging Face Transformers Models (LLMs/VLMs)"]:::external
GPU["GPU / CUDA Runtime"]:::external
OpenAI["OpenAI GPT-4o API (Judge)"]:::external
FS["Filesystem"]:::external

Artifacts["Artifacts (directions, generations, scores, CSVs/JSON)"]:::artifact
Inputs["Inputs (concept datasets, prompt templates)"]:::data

User -->|"runs pipelines/demos"| Repo
Repo <-->|"load weights + run inference (GPU compute)"| HF
Repo -->|"GPU compute"| GPU
Repo <-->|"read/write"| FS
FS -->|"read"| Inputs
Repo -->|"persist"| Artifacts
Repo -->|"network call: judge prompts + generations"| OpenAI
OpenAI -->|"scores + feedback"| Repo
end

%% =======================================
%% Level 2 — Container (Subsystems inside)
%% =======================================
subgraph "Level 2 — Containers (Benchmark vs VLM + Shared Core Library)"
direction TB

subgraph "Steering Benchmark (Batch Pipeline)"
direction TB
SB["steering_benchmark/"]:::container
SB_run["run.py (Direction Computation)"]:::script
SB_gen["eval_generations.py (Steered Generation)"]:::script
SB_judge["parse_results.py (GPT-4o Evaluation)"]:::script
SB_agg["read_csv.py (Results Aggregation)"]:::script

SB -->|"stage 1"| SB_run
SB -->|"stage 2"| SB_gen
SB -->|"stage 3"| SB_judge
SB -->|"stage 4"| SB_agg
end

subgraph "VLM Steering (Interactive / Examples)"
direction TB
VLM["vlm_steering/"]:::container
VLM_main["main.py (VLM demo runner)"]:::script
VLM -->|"entrypoint"| VLM_main
end

subgraph "Core Steering Library (Shared)"
direction TB
CoreA["direction_utils.py (Activation capture / hooks)"]:::core
CoreB["neural_controllers.py (Controller runtime)"]:::core
CoreC["generation_utils.py (Steered decoding)"]:::core

CoreD["control_toolkits.py (Direction strategies registry)"]:::algo
CoreE["rfm.py (RFM implementation)"]:::algo
CoreF["utils.py (Dataset loaders + utilities)"]:::core

DataDir["data/ (Concept datasets / taxonomy)"]:::data
PromptDir["evaluation_prompts/ (Judge prompt templates)"]:::data
end

SB_run -->|"load concept dataset"| CoreF
CoreF -->|"read"| DataDir

SB_run -->|"extract hidden states"| CoreA
SB_run -->|"fit direction (RFM/linear/logistic/PCA/mean-diff)"| CoreD
CoreD -->|"optional: use RFM"| CoreE
CoreA <-->|"forward hooks"| HF

SB_gen -->|"load directions"| Artifacts
SB_gen -->|"generate with steering"| CoreC
CoreC -->|"apply controller"| CoreB
CoreC -->|"capture/target layers"| CoreA
CoreC <-->|"decode tokens (GPU compute)"| HF

SB_judge -->|"load generations"| Artifacts
SB_judge -->|"load prompt templates"| PromptDir
SB_judge -->|"network call"| OpenAI
SB_judge -->|"write scores"| Artifacts

SB_agg -->|"read scored CSVs"| Artifacts
SB_agg -->|"output"| Metrics["Aggregate metrics / report"]:::artifact

VLM_main -->|"generate with steering"| CoreC
VLM_main -->|"apply controller"| CoreB
VLM_main -->|"load model"| HF
VLM_main -->|"optional image input"| Image["Image input (optional)"]:::external
end

Repo -->|"contains"| SB
Repo -->|"contains"| VLM
Repo -->|"contains"| CoreA
Repo -->|"contains"| CoreB
Repo -->|"contains"| CoreC
Repo -->|"contains"| CoreD
Repo -->|"contains"| CoreE
Repo -->|"contains"| CoreF
Repo -->|"contains"| DataDir
Repo -->|"contains"| PromptDir

%% ==================================================
%% Level 3 — Benchmark Pipeline (Staged Artifacts)
%% ==================================================
subgraph "Level 3 — Steering Benchmark Pipeline (Staged Batch Processing)"
direction TB

Stage1["1) Direction Computation\n(steering_benchmark/run.py)"]:::script
DirStore[("Directions store\n(vector files/artifacts)")]:::artifact

Stage2["2) Steered Generation\n(steering_benchmark/eval_generations.py)"]:::script
GenStore[("Generations store\n(text/JSONL/CSV)")]:::artifact

Stage3["3) GPT-4o Evaluation\n(steering_benchmark/parse_results.py)"]:::script
ScoreStore[("Scores store\n(CSV/JSON)")]:::artifact

Stage4["4) Aggregation\n(steering_benchmark/read_csv.py)"]:::script
FinalOut[("Final metrics / report")]:::artifact

Stage1 -->|"save direction vectors"| DirStore
DirStore -->|"load directions"| Stage2
Stage2 -->|"write steered outputs"| GenStore
GenStore -->|"load generations"| Stage3
Stage3 -->|"write scored results"| ScoreStore
ScoreStore -->|"read CSVs"| Stage4
Stage4 -->|"produce summary"| FinalOut

Stage1 -->|"uses"| CoreF
Stage1 -->|"uses"| CoreA
Stage1 -->|"uses"| CoreD
CoreD -->|"uses (when RFM selected)"| CoreE

Stage2 -->|"uses"| CoreC
Stage2 -->|"uses"| CoreB
Stage2 -->|"uses"| CoreA

Stage3 -->|"reads"| PromptDir
Stage3 -->|"calls (network)"| OpenAI
end

%% ===========================================
%% Runtime Steering Loop (Generation Sequence)
%% ===========================================
subgraph "Runtime — Inference-time Steering Loop (Conceptual)"
direction TB

PromptTok["Prompt + Tokenizer"]:::core
Forward["Model forward pass\n(HF Transformers)"]:::external
Hook["Forward hooks capture layer states\n(direction_utils.py)"]:::core
Apply["Controller modifies activations\n(hidden += coef * direction)\n(neural_controllers.py)"]:::core
Decode["Sampling/Decoding step\n(generation_utils.py)"]:::core
OutText["Steered output text"]:::artifact

PromptTok -->|"tokens"| Forward
Forward -->|"hidden states"| Hook
Hook -->|"extracted activations"| Apply
Apply -->|"modified activations"| Forward
Forward -->|"logits"| Decode
Decode -->|"next token"| Forward
Decode -->|"final text"| OutText
end

%% ============
%% Click Events
%% ============
click SB "https://github.com/dmbeaglehole/universal_steering/tree/main/steering_benchmark/"
click SB_run "https://github.com/dmbeaglehole/universal_steering/blob/main/steering_benchmark/run.py"
click SB_gen "https://github.com/dmbeaglehole/universal_steering/blob/main/steering_benchmark/eval_generations.py"
click SB_judge "https://github.com/dmbeaglehole/universal_steering/blob/main/steering_benchmark/parse_results.py"
click SB_agg "https://github.com/dmbeaglehole/universal_steering/blob/main/steering_benchmark/read_csv.py"

click VLM "https://github.com/dmbeaglehole/universal_steering/tree/main/vlm_steering/"
click VLM_main "https://github.com/dmbeaglehole/universal_steering/blob/main/vlm_steering/main.py"

click CoreB "https://github.com/dmbeaglehole/universal_steering/blob/main/neural_controllers.py"
click CoreC "https://github.com/dmbeaglehole/universal_steering/blob/main/generation_utils.py"
click CoreA "https://github.com/dmbeaglehole/universal_steering/blob/main/direction_utils.py"
click CoreD "https://github.com/dmbeaglehole/universal_steering/blob/main/control_toolkits.py"
click CoreE "https://github.com/dmbeaglehole/universal_steering/blob/main/rfm.py"
click CoreF "https://github.com/dmbeaglehole/universal_steering/blob/main/utils.py"
click DataDir "https://github.com/dmbeaglehole/universal_steering/tree/main/data/"
click PromptDir "https://github.com/dmbeaglehole/universal_steering/tree/main/evaluation_prompts/"

%% ======
%% Styles
%% ======
classDef system fill:#0b3d91,stroke:#062c66,color:#ffffff,stroke-width:2px
classDef container fill:#1f6feb,stroke:#0b3d91,color:#ffffff,stroke-width:1.5px
classDef script fill:#2b8cff,stroke:#0b3d91,color:#ffffff,stroke-width:1.5px
classDef core fill:#1f9d55,stroke:#0f6d3a,color:#ffffff,stroke-width:1.5px
classDef algo fill:#f59e0b,stroke:#b45309,color:#111827,stroke-width:1.5px
classDef data fill:#9ca3af,stroke:#4b5563,color:#111827,stroke-width:1.5px
classDef artifact fill:#6b7280,stroke:#374151,color:#ffffff,stroke-width:1.5px
classDef external fill:#7c3aed,stroke:#4c1d95,color:#ffffff,stroke-width:1.5px
```
