# princeton-pli/hal-harness architecture

> Architecture diagram of the GitHub repository princeton-pli/hal-harness, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/princeton-pli/hal-harness
- Repository: https://github.com/princeton-pli/hal-harness

## Overview

##### Project Type and Purpose
- This is an open-source CLI-based evaluation harness (“HAL”) for running and comparing AI agents across multiple benchmarks.
- It’s not a web app but a tooling library/CLI with plugin-style benchmarks and agent implementations.
- Key use cases:
- “hal-eval” CLI to run agents on benchmarks (local/Docker/Azure VM)
- “hal-upload” CLI to encrypt and push results to HuggingFace Hub
- Integration with Weave (wandb.ai), Inspect AI, Azure, Docker, HF Hub

##### High-Level File Structure Analysis
Top-level
- pyproject.toml / .env.template – packaging and environment config
- README.md – describes features, benchmarks, CLI usage
- agents/ – example agents grouped by benchmark (USACO, SWE-bench, Inspect AI, etc.)
- hal/ – core harness code
– cli.py – CLI entrypoint commands (hal-eval, hal-upload, hal-decrypt)
– benchmark_manager.py – selects and orchestrates benchmarks
– agent_runner.py – loads & invokes user agent code
– upload_manager.py – handles result encryption & HF upload
– benchmarks/ – plugin modules implementing each benchmark interface
– inspect/ – integration code for Inspect AI tasks
– utils/ – runners (local, docker_runner, vm_runner), encryption, logging, Weave integration
- tests/ – unit/integration tests

##### Key Components
a. CLI Layer (hal-eval / hal-upload / hal-decrypt)
b. Core Harness
- CLI Parser (hal/cli.py)
- Benchmark Manager (hal/benchmark_manager.py)
- Agent Runner (hal/agent_runner.py)
- Upload Manager (hal/upload_manager.py)
c. Execution Engines
- Local Runner (hal/utils/local_runner.py)
- Docker Runner (hal/utils/docker_runner.py + Dockerfile)
- Azure VM Runner (hal/utils/vm_runner.py + azure_utils.py)
d. Benchmark Modules (hal/benchmarks/*)
e. Agents Directory (User-provided code under agents/)
– Dynamic import and function invocation
f. External Services
- Weave for logging & cost tracking (via hal/utils/weave_utils.py)
- Inspect AI (via hal/inspect/*)
- HuggingFace Hub (via Upload Manager)
- Azure Resource Manager (via azure_utils)
- Docker daemon

##### Component Interactions and Data Flow
1. User issues CLI command:
hal-eval → CLI Parser
↓
Benchmark Manager selects hal/benchmarks/<benchmark>.py
↓
Agent Runner loads agent_dir + agent_function → executes on runner
↙   ↘
Local Runner  Docker Runner  VM Runner
↓              ↓             ↓
Runs agent+benchmark → returns results & traces
↓
Weave Utils logs metrics & costs to Weave
↓
Results passed to Upload Manager when “--upload” → encryption utils → HF Hub API
2. hal-upload / hal-decrypt CLI commands bypass evaluation and talk directly to Upload Manager

##### Architectural Patterns & Design Principles
- Plugin Architecture: benchmarks and agents are pluggable directories
- Strategy Pattern: execution strategy abstracted into runners (local/docker/vm)
- Single Responsibility: separate managers for CLI parsing, benchmarking, agent invocation, uploading
- Separation of Concerns: core harness vs. external integrations vs. utility helpers

##### Technologies and Frameworks
- Python 3.12, pyproject.toml (Poetry/PEP 621)
- Conda environments, Docker, Azure SDK
- Weave (wandb.ai), Inspect AI, HuggingFace Hub APIs
- GPG encryption for results (.gpg files)
- Logging via hal/utils/logging_utils.py

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        CLI["CLI Parser\n(hal/cli.py)"]:::cli
    end

    %% Core Harness
    subgraph "Core Harness"
        BM["Benchmark Manager\n(hal/benchmark_manager.py)"]:::core
        AR["Agent Runner\n(hal/agent_runner.py)"]:::core
        UM["Upload Manager\n(hal/upload_manager.py)"]:::core
    end

    %% Plugins
    subgraph "Plugins"
        subgraph "Benchmarks" 
            US["USACO\n(hal/benchmarks/usaco.py)"]:::plugin
            SB["SWE-bench\n(hal/benchmarks/swebench.py)"]:::plugin
            SC["SciCode\n(hal/benchmarks/scicode.py)"]:::plugin
            GA["GAIA\n(hal/benchmarks/gaia.py)"]:::plugin
            AW["AppWorld\n(hal/benchmarks/appworld.py)"]:::plugin
            CB["ColBench\n(hal/benchmarks/colbench.py)"]:::plugin
            CB2["CoreBench\n(hal/benchmarks/corebench.py)"]:::plugin
            SAB["ScienceAgentBench\n(hal/benchmarks/scienceagentbench.py)"]:::plugin
            TB["TauBench\n(hal/benchmarks/taubench.py)"]:::plugin
            IB["InspectBenchmark\n(hal/benchmarks/inspect_benchmark.py)"]:::plugin
        end
        AG["Agents Directory\n(agents/)"]:::plugin
    end

    %% Execution Engine
    subgraph "Execution Engine"
        LR["Local Runner\n(hal/utils/local_runner.py)"]:::runner
        DR["Docker Runner\n(hal/utils/docker_runner.py)"]:::runner
        VMR["Azure VM Runner\n(hal/utils/vm_runner.py)"]:::runner
    end

    %% Utilities
    subgraph "Utilities"
        ENC["Encryption Utils\n(hal/utils/encryption.py,\nhal/utils/json_encryption.py)"]:::util
        AZU["Azure Utils\n(hal/utils/azure_utils.py)"]:::util
        WU["Weave Utils\n(hal/utils/weave_utils.py)"]:::util
        LU["Logging Utils\n(hal/utils/logging_utils.py)"]:::util
        CU["Core Utils\n(hal/utils/utils.py,\nhal/utils/validation.py)"]:::util
    end

    %% External Systems
    subgraph "External Systems"
        DockerDaemon["Docker Daemon"]:::ext
        WeaveSvc["Weave"]:::ext
        InspectAI["Inspect AI"]:::ext
        HFHub["HuggingFace Hub"]:::ext
        AzureRM["Azure Resource Manager"]:::ext
    end

    %% Connections
    CLI -->|"invoke command"| BM
    BM -->|"selects benchmark"| US & SB & SC & GA & AW & CB & CB2 & SAB & TB & IB
    BM -->|"delegates to"| AR
    AR -->|"agent code & prompts"| LR & DR & VMR
    LR -->|"results & traces"| WU
    DR -->|"results & traces"| WU
    VMR -->|"results & traces"| WU
    WU -->|"logs metrics & costs"| WeaveSvc
    LR -->|"raw results"| UM
    DR -->|"raw results"| UM
    VMR -->|"raw results"| UM
    UM -->|"encrypted payload"| HFHub

    %% CLI upload/decrypt
    CLI -->|"upload/decrypt"| UM

    %% Runner to external infra
    DR -->|"uses"| DockerDaemon
    VMR -->|"uses"| AzureRM
    AR -->|"may integrate"| InspectAI

    %% Click Events
    click CLI "https://github.com/princeton-pli/hal-harness/blob/main/hal/cli.py"
    click BM "https://github.com/princeton-pli/hal-harness/blob/main/hal/benchmark_manager.py"
    click AR "https://github.com/princeton-pli/hal-harness/blob/main/hal/agent_runner.py"
    click UM "https://github.com/princeton-pli/hal-harness/blob/main/hal/upload_manager.py"
    click LR "https://github.com/princeton-pli/hal-harness/blob/main/hal/utils/local_runner.py"
    click DR "https://github.com/princeton-pli/hal-harness/blob/main/hal/utils/docker_runner.py"
    click VMR "https://github.com/princeton-pli/hal-harness/blob/main/hal/utils/vm_runner.py"
    click ENC "https://github.com/princeton-pli/hal-harness/blob/main/hal/utils/encryption.py"
    click ENC "https://github.com/princeton-pli/hal-harness/blob/main/hal/utils/json_encryption.py"
    click AZU "https://github.com/princeton-pli/hal-harness/blob/main/hal/utils/azure_utils.py"
    click WU "https://github.com/princeton-pli/hal-harness/blob/main/hal/utils/weave_utils.py"
    click LU "https://github.com/princeton-pli/hal-harness/blob/main/hal/utils/logging_utils.py"
    click CU "https://github.com/princeton-pli/hal-harness/blob/main/hal/utils/utils.py"
    click CU "https://github.com/princeton-pli/hal-harness/blob/main/hal/utils/validation.py"
    click US "https://github.com/princeton-pli/hal-harness/blob/main/hal/benchmarks/usaco.py"
    click SB "https://github.com/princeton-pli/hal-harness/blob/main/hal/benchmarks/swebench.py"
    click SC "https://github.com/princeton-pli/hal-harness/blob/main/hal/benchmarks/scicode.py"
    click GA "https://github.com/princeton-pli/hal-harness/blob/main/hal/benchmarks/gaia.py"
    click AW "https://github.com/princeton-pli/hal-harness/blob/main/hal/benchmarks/appworld.py"
    click CB "https://github.com/princeton-pli/hal-harness/blob/main/hal/benchmarks/colbench.py"
    click CB2 "https://github.com/princeton-pli/hal-harness/blob/main/hal/benchmarks/corebench.py"
    click SAB "https://github.com/princeton-pli/hal-harness/blob/main/hal/benchmarks/scienceagentbench.py"
    click TB "https://github.com/princeton-pli/hal-harness/blob/main/hal/benchmarks/taubench.py"
    click IB "https://github.com/princeton-pli/hal-harness/blob/main/hal/benchmarks/inspect_benchmark.py"
    click AG "https://github.com/princeton-pli/hal-harness/tree/main/agents/"
    click DockerDaemon "https://github.com/princeton-pli/hal-harness/tree/main/hal/utils/docker/Dockerfile"

    %% Styles
    classDef cli fill:#D6EFFF,stroke:#0366d6
    classDef core fill:#CCE5FF,stroke:#004BA0
    classDef plugin fill:#F0F0F0,stroke:#6A6A6A
    classDef runner fill:#E2F8E2,stroke:#28A745
    classDef util fill:#FFF2CC,stroke:#BF8F00
    classDef ext fill:#FFEBCD,stroke:#D2691E
```
