How harbor-framework/harbor works
Harbor is a framework for evaluating agents and language models in sandboxed benchmark environments, creating benchmarks, running parallel experiments, and generating RL rollouts. Its principal workflow is to select a dataset and agent/model, execute tasks in an environment, score results, and inspect or upload runs. The graph preserves these documented capabilities and the registry, viewer, and upload subsystems. Evidence is limited to the README, the source index, and sampled excerpts; unverified runtime links are omitted or marked dashed.
Main entry points: cli Harbor CLI · server.py Viewer API · json.py Dataset registry · agents Agent adapters
Components 17
Evaluation runs
- Harbor CLI
src/harbor/cli - Agent adapters
src/harbor/agents
Datasets and tasks
- Task configuration
src/harbor/models/task/config.py - Benchmark adapters
adapters - Dataset commands
src/harbor/cli/datasets.py - Dataset registry
src/harbor/registry/client/json.py
Runtime infrastructure
- Sandbox environments
src/harbor - Environment providers
Results and analysis
- Scoring and metrics
adapters - Rollout generation
src/harbor - Run results
src/harbor/db - Viewer API
src/harbor/viewer/server.py - Viewer UI
apps/viewer/app - Run upload
src/harbor/upload
Other components
- Engineer
- Harbor Hub
- Model services
Connections 18
- Engineer → Harbor CLI: starts run
README.md - Harbor CLI → Dataset registry: selects dataset
- Dataset commands → Dataset registry: lists datasets
src/harbor/cli/datasets.py - Dataset registry → Benchmark adapters: describes datasets
adapters/bird_bench/src/bird_bench/main.py - Harbor CLI → Agent adapters: selects agent
- Harbor CLI → Task configuration: configures tasks
- Benchmark adapters → Task configuration: defines task inputs
adapters/frontier-cs-algorithm/src/frontier_cs_algorithm/adapter.py - Agent adapters → Model services: requests model output
- Harbor CLI → Sandbox environments: launches tasks
README.md - Sandbox environments → Environment providers: runs on provider
README.md - Harbor CLI → Scoring and metrics: evaluates outputs
adapters/bird_bench/src/bird_bench/main.py - Harbor CLI → Rollout generation: generates rollouts
README.md - Harbor CLI → Run results: records run
src/harbor/cli/datasets.py - Viewer UI → Viewer API: requests run data
- Viewer API → Run results: reads run data
src/harbor/viewer/server.py - Viewer API → Harbor Hub: checks upload status
src/harbor/viewer/server.py - Run upload → Harbor Hub: transfers run files
src/harbor/upload/storage.py - Engineer → Viewer UI: inspects results