How princeton-pli/hal-harness works
Project Type and Purpose
• This is an open-source CLI-based evaluation harness (“HAL”) for running and comparing AI agents across multiple benchmarks.
• It’s not a web app but a tooling library/CLI with plugin-style benchmarks and agent implementations.
• Key use cases:
• “hal-eval” CLI to run agents on benchmarks (local/Docker/Azure VM)
• “hal-upload” CLI to encrypt and push results to HuggingFace Hub
• Integration with Weave (wandb.ai), Inspect AI, Azure, Docker, HF Hub
High-Level File Structure Analysis
Top-level
• pyproject.toml / .env.template – packaging and environment config
• README.md – describes features, benchmarks, CLI usage
• agents/ – example agents grouped by benchmark (USACO, SWE-bench, Inspect AI, etc.)
• hal/ – core harness code
– cli.py – CLI entrypoint commands (hal-eval, hal-upload, hal-decrypt)
– benchmark_manager.py – selects and orchestrates benchmarks
– agent_runner.py – loads & invokes user agent code
– upload_manager.py – handles result encryption & HF upload
– benchmarks/ – plugin modules implementing each benchmark interface
– inspect/ – integration code for Inspect AI tasks
– utils/ – runners (local, docker_runner, vm_runner), encryption, logging, Weave integration
• tests/ – unit/integration tests
Key Components
a. CLI Layer (hal-eval / hal-upload / hal-decrypt)
b. Core Harness
• CLI Parser (hal/cli.py)
• Benchmark Manager (hal/benchmark_manager.py)
• Agent Runner (hal/agent_runner.py)
• Upload Manager (hal/upload_manager.py)
c. Execution Engines
• Local Runner (hal/utils/local_runner.py)
• Docker Runner (hal/utils/docker_runner.py + Dockerfile)
• Azure VM Runner (hal/utils/vm_runner.py + azure_utils.py)
d. Benchmark Modules (hal/benchmarks/*)
e. Agents Directory (User-provided code under agents/)
– Dynamic import and function invocation
f. External Services
• Weave for logging & cost tracking (via hal/utils/weave_utils.py)
• Inspect AI (via hal/inspect/*)
• HuggingFace Hub (via Upload Manager)
• Azure Resource Manager (via azure_utils)
• Docker daemon
Component Interactions and Data Flow
1. User issues CLI command:
hal-eval → CLI Parser
↓
Benchmark Manager selects hal/benchmarks/<benchmark>.py
↓
Agent Runner loads agent_dir + agent_function → executes on runner
↙ ↘
Local Runner Docker Runner VM Runner
↓ ↓ ↓
Runs agent+benchmark → returns results & traces
↓
Weave Utils logs metrics & costs to Weave
↓
Results passed to Upload Manager when “--upload” → encryption utils → HF Hub API
2. hal-upload / hal-decrypt CLI commands bypass evaluation and talk directly to Upload Manager
Architectural Patterns & Design Principles
• Plugin Architecture: benchmarks and agents are pluggable directories
• Strategy Pattern: execution strategy abstracted into runners (local/docker/vm)
• Single Responsibility: separate managers for CLI parsing, benchmarking, agent invocation, uploading
• Separation of Concerns: core harness vs. external integrations vs. utility helpers
Technologies and Frameworks
• Python 3.12, pyproject.toml (Poetry/PEP 621)
• Conda environments, Docker, Azure SDK
• Weave (wandb.ai), Inspect AI, HuggingFace Hub APIs
• GPG encryption for results (.gpg files)
• Logging via hal/utils/logging_utils.py