# rogo-technologies/big-finance-benchmark architecture

> Architecture diagram of the GitHub repository rogo-technologies/big-finance-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rogo-technologies/big-finance-benchmark
- Repository: https://github.com/rogo-technologies/big-finance-benchmark

## Overview

Big Finance Harness is a reference scaffold for evaluating LLM agents on financial-research questions. The principal workflow loads benchmark items, runs each question through a ReAct agent using a LiteLLM-backed model and research tools, records traces, then grades them against expert rubrics. The orchestrator also supports resumption; the README documents subsequent cost recomputation and analysis/reporting. Tool and model internals are represented where the excerpts or source index establish their roles; unsampled implementation details are not inferred.

Main entry points: [`big_finance_harness/tools/base.py`](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/base.py) (Tool interface), [`big_finance_harness/agent.py`](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/agent.py) (ReAct agent), [`scripts/run_eval_set.py`](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/scripts/run_eval_set.py) (Run orchestrator), [`big_finance_harness/trace.py`](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/trace.py) (Run traces)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Evaluation orchestration"]
  node_orchestrator["Run orchestrator<br/>[run_eval_set.py]"]
  node_resume["Resume policy<br/>[resumption.py]"]
  node_costs["Cost recomputation<br/>[recompute_costs.py]"]
  node_analysis["Analysis tables"]
  node_headline["Headline metrics<br/>[headline_table.py]"]
  node_plots["Result plots<br/>[build_plots.py]"]
end

subgraph group_agent_runtime["Agent runtime"]
  node_agent["ReAct agent<br/>[agent.py]"]
  node_model_client["Model client<br/>[base.py]"]
  node_message_types["Message and run types<br/>[types.py]"]
  node_tool_base["Tool interface<br/>[base.py]"]
end

subgraph group_research_tools["Research tools"]
  node_web_search["Web search<br/>[web_search.py]"]
  node_edgar_search["SEC EDGAR search<br/>[edgar_search.py]"]
  node_fetch_url["Fetch documents<br/>[fetch_url.py]"]
  node_blocked_urls["URL blocking<br/>[blocked_urls.py]"]
  node_python_exec["Python execution<br/>[python_exec.py]"]
  node_final_answer["Final answer tool<br/>[final_answer.py]"]
end

subgraph group_evaluation_data["Benchmark and results"]
  node_dataset[("Benchmark dataset")]
  node_trace_store[("Run traces<br/>[trace.py]")]
  node_grader["Rubric grader<br/>[grader.py]"]
  node_grade_store[("Grade outputs")]
end

node_operator(("Evaluator"))
node_search_provider["Search provider"]
node_sec_edgar["SEC EDGAR"]
node_web_documents["Web documents"]

node_operator -->|"starts run"| node_orchestrator
node_dataset -->|"supplies items"| node_orchestrator
node_orchestrator -->|"runs questions"| node_agent
node_orchestrator -->|"grades runs"| node_grader
node_orchestrator -->|"checks completion"| node_resume
node_agent -->|"requests completions"| node_model_client
node_agent -->|"dispatches tools"| node_tool_base
node_agent -->|"uses messages"| node_message_types
node_model_client -->|"translates messages"| node_message_types
node_web_search -->|"implements interface"| node_tool_base
node_edgar_search -->|"implements interface"| node_tool_base
node_fetch_url -->|"implements interface"| node_tool_base
node_python_exec -->|"implements interface"| node_tool_base
node_final_answer -->|"implements interface"| node_tool_base
node_web_search -->|"filters sources"| node_blocked_urls
node_fetch_url -->|"checks URL"| node_blocked_urls
node_web_search -->|"queries"| node_search_provider
node_edgar_search -->|"queries filings"| node_sec_edgar
node_fetch_url -->|"fetches content"| node_web_documents
node_agent -->|"records run"| node_trace_store
node_trace_store -->|"reads traces"| node_resume
node_trace_store -->|"provides run"| node_grader
node_grader -->|"requests judge call"| node_model_client
node_grader -->|"produces grades"| node_grade_store
node_grade_store -->|"checks completed grades"| node_resume
node_trace_store -->|"supplies traces"| node_costs
node_grade_store -->|"supplies grades"| node_analysis
node_analysis -->|"feeds metrics"| node_headline
node_analysis -->|"feeds plots"| node_plots

click node_orchestrator "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/scripts/run_eval_set.py"
click node_resume "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/resumption.py"
click node_costs "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/scripts/recompute_costs.py"
click node_analysis "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/scripts/build_analysis_csv.py"
click node_headline "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/scripts/headline_table.py"
click node_plots "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/scripts/build_plots.py"
click node_agent "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/agent.py"
click node_model_client "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/models/base.py"
click node_message_types "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/types.py"
click node_tool_base "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/base.py"
click node_web_search "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/web_search.py"
click node_edgar_search "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/edgar_search.py"
click node_fetch_url "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/fetch_url.py"
click node_blocked_urls "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/blocked_urls.py"
click node_python_exec "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/python_exec.py"
click node_final_answer "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/final_answer.py"
click node_dataset "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/data/big_finance_subset.jsonl"
click node_trace_store "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/trace.py"
click node_grader "https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/grader.py"
click node_grade_store "https://github.com/rogo-technologies/big-finance-benchmark/tree/main/grades"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orchestrator,node_resume,node_costs,node_analysis,node_headline,node_plots toneBlue
class node_agent,node_model_client,node_message_types,node_tool_base toneAmber
class node_web_search,node_edgar_search,node_fetch_url,node_blocked_urls,node_python_exec,node_final_answer toneMint
class node_dataset,node_trace_store,node_grader,node_grade_store toneRose
class node_operator,node_search_provider,node_sec_edgar,node_web_documents toneIndigo
```

## Components

### Evaluation orchestration

- [Run orchestrator](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/scripts/run_eval_set.py): `scripts/run_eval_set.py`
- [Resume policy](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/resumption.py): `big_finance_harness/resumption.py`
- [Cost recomputation](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/scripts/recompute_costs.py): `scripts/recompute_costs.py`
- [Analysis tables](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/scripts/build_analysis_csv.py): `scripts/build_analysis_csv.py`
- [Headline metrics](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/scripts/headline_table.py): `scripts/headline_table.py`
- [Result plots](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/scripts/build_plots.py): `scripts/build_plots.py`

### Agent runtime

- [ReAct agent](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/agent.py): `big_finance_harness/agent.py`
- [Model client](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/models/base.py): `big_finance_harness/models/base.py`
- [Message and run types](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/types.py): `big_finance_harness/types.py`
- [Tool interface](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/base.py): `big_finance_harness/tools/base.py`

### Research tools

- [Web search](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/web_search.py): `big_finance_harness/tools/web_search.py`
- [SEC EDGAR search](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/edgar_search.py): `big_finance_harness/tools/edgar_search.py`
- [Fetch documents](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/fetch_url.py): `big_finance_harness/tools/fetch_url.py`
- [URL blocking](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/blocked_urls.py): `big_finance_harness/tools/blocked_urls.py`
- [Python execution](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/python_exec.py): `big_finance_harness/tools/python_exec.py`
- [Final answer tool](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/tools/final_answer.py): `big_finance_harness/tools/final_answer.py`

### Benchmark and results

- [Benchmark dataset](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/data/big_finance_subset.jsonl): `data/big_finance_subset.jsonl`
- [Run traces](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/trace.py): `big_finance_harness/trace.py`
- [Rubric grader](https://github.com/rogo-technologies/big-finance-benchmark/blob/main/big_finance_harness/grader.py): `big_finance_harness/grader.py`
- [Grade outputs](https://github.com/rogo-technologies/big-finance-benchmark/tree/main/grades): `grades`

### Other components

- **Evaluator**
- **Search provider**
- **SEC EDGAR**
- **Web documents**

## Connections

- Evaluator → Run orchestrator: starts run (evidence: `README.md`)
- Benchmark dataset → Run orchestrator: supplies items (evidence: `README.md`)
- Run orchestrator → ReAct agent: runs questions (evidence: `scripts/run_eval_set.py`)
- Run orchestrator → Rubric grader: grades runs (evidence: `scripts/run_eval_set.py`)
- Run orchestrator → Resume policy: checks completion (evidence: `big_finance_harness/resumption.py`)
- ReAct agent → Model client: requests completions (evidence: `big_finance_harness/agent.py`)
- ReAct agent → Tool interface: dispatches tools (evidence: `big_finance_harness/agent.py`)
- ReAct agent → Message and run types: uses messages (evidence: `big_finance_harness/agent.py`)
- Model client → Message and run types: translates messages (evidence: `big_finance_harness/models/base.py`)
- Web search → Tool interface: implements interface (evidence: `big_finance_harness/tools/web_search.py`)
- SEC EDGAR search → Tool interface: implements interface (evidence: `big_finance_harness/tools/edgar_search.py`)
- Fetch documents → Tool interface: implements interface (evidence: `big_finance_harness/tools/fetch_url.py`)
- Python execution → Tool interface: implements interface (evidence: `big_finance_harness/tools/base.py`)
- Final answer tool → Tool interface: implements interface (evidence: `big_finance_harness/tools/final_answer.py`)
- Web search → URL blocking: filters sources (evidence: `big_finance_harness/tools/web_search.py`)
- Fetch documents → URL blocking: checks URL (evidence: `big_finance_harness/tools/fetch_url.py`)
- Web search → Search provider: queries (evidence: `README.md`)
- SEC EDGAR search → SEC EDGAR: queries filings (evidence: `README.md`)
- Fetch documents → Web documents: fetches content (evidence: `README.md`)
- ReAct agent → Run traces: records run (evidence: `README.md`)
- Run traces → Resume policy: reads traces (evidence: `big_finance_harness/resumption.py`)
- Run traces → Rubric grader: provides run (evidence: `big_finance_harness/grader.py`)
- Rubric grader → Model client: requests judge call (evidence: `big_finance_harness/grader.py`)
- Rubric grader → Grade outputs: produces grades (evidence: `README.md`)
- Grade outputs → Resume policy: checks completed grades (evidence: `big_finance_harness/resumption.py`)
- Run traces → Cost recomputation: supplies traces (evidence: `README.md`)
- Grade outputs → Analysis tables: supplies grades (evidence: `README.md`)
- Analysis tables → Headline metrics: feeds metrics (evidence: `README.md`)
- Analysis tables → Result plots: feeds plots (evidence: `README.md`)
