How rogo-technologies/big-finance-benchmark works
Big Finance Harness is a reference scaffold for evaluating LLM agents on financial-research questions. The principal workflow loads benchmark items, runs each question through a ReAct agent using a LiteLLM-backed model and research tools, records traces, then grades them against expert rubrics. The orchestrator also supports resumption; the README documents subsequent cost recomputation and analysis/reporting. Tool and model internals are represented where the excerpts or source index establish their roles; unsampled implementation details are not inferred.
Main entry points: base.py Tool interface · agent.py ReAct agent · run_eval_set.py Run orchestrator · trace.py Run traces
Components 24
Evaluation orchestration
- Run orchestrator
scripts/run_eval_set.py - Resume policy
big_finance_harness/resumption.py - Cost recomputation
scripts/recompute_costs.py - Analysis tables
scripts/build_analysis_csv.py - Headline metrics
scripts/headline_table.py - Result plots
scripts/build_plots.py
Agent runtime
- ReAct agent
big_finance_harness/agent.py - Model client
big_finance_harness/models/base.py - Message and run types
big_finance_harness/types.py - Tool interface
big_finance_harness/tools/base.py
Research tools
- Web search
big_finance_harness/tools/web_search.py - SEC EDGAR search
big_finance_harness/tools/edgar_search.py - Fetch documents
big_finance_harness/tools/fetch_url.py - URL blocking
big_finance_harness/tools/blocked_urls.py - Python execution
big_finance_harness/tools/python_exec.py - Final answer tool
big_finance_harness/tools/final_answer.py
Benchmark and results
- Benchmark dataset
data/big_finance_subset.jsonl - Run traces
big_finance_harness/trace.py - Rubric grader
big_finance_harness/grader.py - Grade outputs
grades
Other components
- Evaluator
- Search provider
- SEC EDGAR
- Web documents
Connections 29
- Evaluator → Run orchestrator: starts run
README.md - Benchmark dataset → Run orchestrator: supplies items
README.md - Run orchestrator → ReAct agent: runs questions
scripts/run_eval_set.py - Run orchestrator → Rubric grader: grades runs
scripts/run_eval_set.py - Run orchestrator → Resume policy: checks completion
big_finance_harness/resumption.py - ReAct agent → Model client: requests completions
big_finance_harness/agent.py - ReAct agent → Tool interface: dispatches tools
big_finance_harness/agent.py - ReAct agent → Message and run types: uses messages
big_finance_harness/agent.py - Model client → Message and run types: translates messages
big_finance_harness/models/base.py - Web search → Tool interface: implements interface
big_finance_harness/tools/web_search.py - SEC EDGAR search → Tool interface: implements interface
big_finance_harness/tools/edgar_search.py - Fetch documents → Tool interface: implements interface
big_finance_harness/tools/fetch_url.py - Python execution → Tool interface: implements interface
big_finance_harness/tools/base.py - Final answer tool → Tool interface: implements interface
big_finance_harness/tools/final_answer.py - Web search → URL blocking: filters sources
big_finance_harness/tools/web_search.py - Fetch documents → URL blocking: checks URL
big_finance_harness/tools/fetch_url.py - Web search → Search provider: queries
README.md - SEC EDGAR search → SEC EDGAR: queries filings
README.md - Fetch documents → Web documents: fetches content
README.md - ReAct agent → Run traces: records run
README.md - Run traces → Resume policy: reads traces
big_finance_harness/resumption.py - Run traces → Rubric grader: provides run
big_finance_harness/grader.py - Rubric grader → Model client: requests judge call
big_finance_harness/grader.py - Rubric grader → Grade outputs: produces grades
README.md - Grade outputs → Resume policy: checks completed grades
big_finance_harness/resumption.py - Run traces → Cost recomputation: supplies traces
README.md - Grade outputs → Analysis tables: supplies grades
README.md - Analysis tables → Headline metrics: feeds metrics
README.md - Analysis tables → Result plots: feeds plots
README.md