How harveyai/harvey-labs works
Harvey LAB benchmarks legal-work agents using task instructions, documents, and rubrics. Its main workflow is to load a task, run an agent through a provider adapter and bounded workspace tools, then score the run and present results. The repository also includes task discovery, document-production skills, trajectory playback, and model sweeps. Some orchestration and evaluation source was not sampled, so uncertain internal links are omitted; unsampled skill implementations are represented by their existing directories.
Main entry points: run.py Run entrypoint · root.py Benchmark root · list_tasks.py Task discovery · agent_loop.py Agent loop
Components 22
Task access
- Task discovery
lab_core/utils/list_tasks.py - Benchmark root
lab_core/root.py
Agent execution
- Run entrypoint
lab_core/harness/run.py - Agent loop
lab_core/harness/agent_loop.py - Provider adapters
lab_core/harness/adapters - Workspace tools
lab_core/harness/tools.py - Task sandbox
lab_core/sandbox/sandbox.py - Document parser
lab_core/sandbox/parsers/parse_doc.py - DOCX skills
lab_core/harness/skills/docx/scripts - PPTX skills
lab_core/harness/skills/pptx/scripts - XLSX skills
lab_core/harness/skills/xlsx/scripts
Evaluation
- Evaluation runner
lab_core/evaluation/run_eval.py - Rubric scoring
lab_core/evaluation/scoring.py - LLM judge
lab_core/evaluation/judge.py - Run report
lab_core/evaluation/report.py - Run comparison
lab_core/evaluation/compare.py - Charts
lab_core/evaluation/charts.py
Review tools
- Trajectory playback
lab_core/utils/playback.py - Model sweeps
lab_core/utils/sweep.py
Other components
- Benchmark user
- Task dataset
- Model providers
Connections 21
- Benchmark user → Run entrypoint: starts run
- Benchmark user → Task discovery: lists tasks
- Run entrypoint → Benchmark root: resolves root
- Task discovery → Benchmark root: resolves root
- Run entrypoint → Task dataset: loads task
- Task discovery → Task dataset: scans tasks
- Run entrypoint → Provider adapters: creates adapter
- Provider adapters → Model providers: calls API
- Run entrypoint → Agent loop: starts loop
- Agent loop → Provider adapters: requests response
- Agent loop → Workspace tools: dispatches tools
- Workspace tools → Task sandbox: executes workspace
- Task sandbox → Document parser: runs parser
- Evaluation runner → Rubric scoring: scores run
- Rubric scoring → LLM judge: requests judgments
- Run report → Benchmark root: resolves results
- Run comparison → Benchmark root: reads results
- Run comparison → Charts: builds charts
- Trajectory playback → Benchmark root: loads run data
- Model sweeps → Run entrypoint: launches runs
- Model sweeps → Evaluation runner: evaluates runs