# patronus-ai/financebench architecture

> Architecture diagram of the GitHub repository patronus-ai/financebench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/patronus-ai/financebench
- Repository: https://github.com/patronus-ai/financebench

## Overview

##### Project Type and Purpose
- This is an open‑source benchmark and evaluation toolkit (“FinanceBench”) for financial question answering using large language models (LLMs). It provides:
- A small public dataset (150 examples) in JSONL for questions, answers, evidence, and document metadata.
- A corpus of source PDFs for each company and period.
- Precomputed model outputs in JSONL under `/results/`.
- A Jupyter notebook (`evaluation_playground.ipynb`) demonstrating how to load data, perform retrieval, invoke LLMs, and compute metrics.
- There is no frontend or backend server; this is a data + pipeline + evaluation codebase, i.e. an open‑source tool for LLM QA benchmarking.

##### File Structure
- Top‑level files & directories:
- `README.md` – overview, architecture notes.
- `/data/` – JSONL files: questions+evidence and document metadata.
- `/pdfs/` – raw PDF documents for retrieval.
- `/results/` – JSONL model outputs for various configurations.
- `/vectorstores/` – likely store indices or scripts to build them.
- `evaluation_playground.ipynb` – orchestrator for loading data, building retrieval index, issuing prompts to LLMs, and evaluating.
- No MVC or microservices; rather an ETL + retrieval + evaluation pipeline.

##### Extract Insights from README
- Workflow described:
1. Load `financebench_open_source.jsonl` and `financebench_document_information.jsonl` into pandas and merge.
2. Retrieve relevant PDF text (full‑page & sentence extracts).
3. Build or load vector store for retrieval (long context vs. single‑store).
4. Feed retrieved evidence + question to LLMs (OpenAI, Llama2, Claude2 via API).
5. Collect and store completions in `/results/`.
6. Manually review and compute error rates / metrics.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Notebook Orchestration
    Notebook["Jupyter Notebook<br/>(evaluation_playground.ipynb)"]:::notebook

    %% Data Storage
    Q1["financebench_open_source.jsonl"]:::storage
    Q2["financebench_document_information.jsonl"]:::storage
    PDF["PDF Repository"]:::storage
    VS["Vector Store"]:::storage
    Result["Result Store"]:::storage

    %% Compute Modules
    ETL["ETL & Preprocessing Module"]:::compute
    Retrieval["Retrieval Module"]:::compute
    LLM["LLM Invocation Module"]:::compute
    Evaluation["Evaluation Module"]:::compute

    %% External Services
    OA["OpenAI API"]:::external
    LL2["Llama2 API"]:::external
    CL["Claude2 API"]:::external

    %% Workflow
    Notebook -->|"loads data"| ETL
    Q1 --> ETL
    Q2 --> ETL

    ETL -->|"merged DataFrame"| Retrieval
    PDF --> Retrieval
    Retrieval -->|"read/write index"| VS
    VS --> Retrieval
    Retrieval -->|"retrieved evidence + question"| LLM

    LLM -->|"API call"| OA
    LLM -->|"API call"| LL2
    LLM -->|"API call"| CL

    LLM -->|"model response"| Result
    Result -->|"gold vs. output"| Evaluation
    Q1 --> Evaluation
    Q2 --> Evaluation

    Notebook -->|"orchestrates retrieval"| Retrieval
    Notebook -->|"orchestrates inference"| LLM
    Notebook -->|"orchestrates evaluation"| Evaluation

    %% Click Events
    click Q1 "https://github.com/patronus-ai/financebench/blob/main/data/financebench_open_source.jsonl"
    click Q2 "https://github.com/patronus-ai/financebench/blob/main/data/financebench_document_information.jsonl"
    click PDF "https://github.com/patronus-ai/financebench/tree/main/pdfs/"
    click Result "https://github.com/patronus-ai/financebench/tree/main/results/"
    click Notebook "https://github.com/patronus-ai/financebench/blob/main/evaluation_playground.ipynb"
    click VS "https://github.com/patronus-ai/financebench/tree/main/vectorstores/"
    click ETL "https://github.com/patronus-ai/financebench/blob/main/evaluation_playground.ipynb"
    click Retrieval "https://github.com/patronus-ai/financebench/blob/main/evaluation_playground.ipynb"
    click LLM "https://github.com/patronus-ai/financebench/blob/main/evaluation_playground.ipynb"
    click Evaluation "https://github.com/patronus-ai/financebench/blob/main/evaluation_playground.ipynb"

    %% Styles
    classDef storage fill:#blue,stroke:#000,color:#fff;
    classDef compute fill:#2ecc71,stroke:#000,color:#000;
    classDef external fill:#95a5a6,stroke:#000,color:#000;
    classDef notebook fill:#f39c12,stroke:#000,color:#000;
```
