How patronus-ai/financebench works
Project Type and Purpose
• This is an open‑source benchmark and evaluation toolkit (“FinanceBench”) for financial question answering using large language models (LLMs). It provides:
• A small public dataset (150 examples) in JSONL for questions, answers, evidence, and document metadata.
• A corpus of source PDFs for each company and period.
• Precomputed model outputs in JSONL under /results/.
• A Jupyter notebook (evaluation_playground.ipynb) demonstrating how to load data, perform retrieval, invoke LLMs, and compute metrics.
• There is no frontend or backend server; this is a data + pipeline + evaluation codebase, i.e. an open‑source tool for LLM QA benchmarking.
File Structure
• Top‑level files & directories:
• README.md – overview, architecture notes.
• /data/ – JSONL files: questions+evidence and document metadata.
• /pdfs/ – raw PDF documents for retrieval.
• /results/ – JSONL model outputs for various configurations.
• /vectorstores/ – likely store indices or scripts to build them.
• evaluation_playground.ipynb – orchestrator for loading data, building retrieval index, issuing prompts to LLMs, and evaluating.
• No MVC or microservices; rather an ETL + retrieval + evaluation pipeline.
Extract Insights from README
• Workflow described:
1. Load financebench_open_source.jsonl and financebench_document_information.jsonl into pandas and merge.
2. Retrieve relevant PDF text (full‑page & sentence extracts).
3. Build or load vector store for retrieval (long context vs. single‑store).
4. Feed retrieved evidence + question to LLMs (OpenAI, Llama2, Claude2 via API).
5. Collect and store completions in /results/.
6. Manually review and compute error rates / metrics.