How colbyrobbins/xai-project-2 works
VLM-SteerHarness is runtime middleware for multimodal model inference with latent-risk monitoring, conditional activation steering, and governance traces. The principal workflow accepts an image/text request, runs a diagnostic model pass, probes captured activations, applies any planned intervention during generation, and returns output plus an audit record. Separate branches support interactive control, concept-vector calibration, and quantitative evaluation. README lifecycle claims fill some gaps; source excerpts are partial, so uncertain implementation links are omitted or dashed.
Main entry points: core.py Harness Runtime · calibration.py Calibration Pipeline · gatekeeper.py Decision Gatekeeper · benchmark.py Benchmark Logic
Components 19
Inference Runtime
- Harness Runtime
src/harness/core.py - VLM Wrapper
src/models/wrapper.py - Runtime Config
src/harness/config.py - Activation Hooks
src/steering/hook.py
Observability & Policy
- Decision Gatekeeper
src/harness/gatekeeper.py - Latent Probes
src/attribution/probes.py - Neuron Profiler
src/attribution/neurons.py - Audit Telemetry
src/harness/telemetry.py
Concept Steering
- Concept Vectors
src/steering/vectors.py - Vector Extractor
src/steering/extractor.py - Calibration Pipeline
src/steering/calibration.py - Calibration Pairs
data/calibration - Calibrated Vectors
data/concepts
Control & Evaluation
- Control Center
app/app.py - Benchmark Runner
eval/run_benchmarks.py - Benchmark Logic
src/eval/benchmark.py - Evaluation Metrics
src/eval/metrics.py - Benchmark Results
eval/benchmark_results.json
Other components
- Request User
Connections 22
- Request User → Harness Runtime: submits request
README.md - Harness Runtime → Runtime Config: uses settings
src/harness/core.py - Harness Runtime → VLM Wrapper: runs inference
src/harness/core.py - Harness Runtime → Activation Hooks: registers hooks
src/harness/core.py - Harness Runtime → Decision Gatekeeper: requests decisions
src/harness/core.py - Decision Gatekeeper → Latent Probes: evaluates activations
src/harness/gatekeeper.py - Decision Gatekeeper → Concept Vectors: retrieves concepts
src/harness/gatekeeper.py - Activation Hooks → Concept Vectors: injects direction
src/steering/hook.py - Harness Runtime → Neuron Profiler: profiles circuits
src/harness/core.py - Harness Runtime → Audit Telemetry: records trace
README.md - Latent Probes → Audit Telemetry: produces probe records
src/attribution/probes.py - Neuron Profiler → Audit Telemetry: produces attributions
src/attribution/neurons.py - Calibration Pipeline → Vector Extractor: uses algorithms
src/steering/calibration.py - Calibration Pipeline → Concept Vectors: creates vectors
src/steering/calibration.py - Calibration Pipeline → VLM Wrapper: extracts activations
src/steering/calibration.py - Calibration Pairs → Calibration Pipeline: calibrates from pairs
- Calibrated Vectors → Harness Runtime: registers concepts
- Request User → Control Center: operates controls
- Control Center → Harness Runtime: submits inference
- Benchmark Runner → Benchmark Logic: runs benchmarks
eval/run_benchmarks.py - Benchmark Logic → Evaluation Metrics: computes measures
- Benchmark Logic → Benchmark Results: records results