# run-llama/parsebench architecture

> Architecture diagram of the GitHub repository run-llama/parsebench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/run-llama/parsebench
- Repository: https://github.com/run-llama/parsebench

## Overview

ParseBench is a document-parsing benchmark: users select a pipeline, run document inference against a parsing provider, evaluate normalized results against the benchmark’s ground truth, and inspect reports or leaderboard comparisons. Its five documented dimensions—tables, charts, content faithfulness, semantic formatting, and visual grounding—are represented in evaluation. The repository also includes a visual-grounding browser and an annotator for inspecting or editing page-level evidence. The README establishes the benchmark workflow; only selected runtime sources were sampled, so uncertain internal wiring is omitted.

Main entry points: [`src/parse_bench/cli.py`](https://github.com/run-llama/parsebench/blob/main/src/parse_bench/cli.py) (CLI), [`src/parse_bench/inference/runner.py`](https://github.com/run-llama/parsebench/blob/main/src/parse_bench/inference/runner.py) (Inference runner), [`src/parse_bench/evaluation/runner.py`](https://github.com/run-llama/parsebench/blob/main/src/parse_bench/evaluation/runner.py) (Evaluation runner), [`src/parse_bench/inference/providers/parse`](https://github.com/run-llama/parsebench/tree/main/src/parse_bench/inference/providers/parse) (Parsing providers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_benchmark["Benchmark workflow"]
  node_cli["CLI<br/>[cli.py]"]
  node_dataset["Dataset download<br/>[download.py]"]
end

subgraph group_inference["Inference"]
  node_pipeline_spec["Pipeline specification<br/>[pipeline.py]"]
  node_inference_runner["Inference runner<br/>[runner.py]"]
  node_provider_registry["Provider registry<br/>[registry.py]"]
  node_provider_adapters["Parsing providers"]
  node_result_models["Result models<br/>[pipeline_io.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_evaluation_runner["Evaluation runner<br/>[runner.py]"]
  node_metric_families["Dimension metrics"]
  node_ground_truth["Ground-truth cases"]
  node_evaluation_reports["Evaluation reports"]
  node_analysis["Score analysis"]
end

subgraph group_grounding_tools["Grounding tools"]
  node_viewer_frontend["Grounding viewer<br/>[App.tsx]"]
  node_viewer_api["Viewer API<br/>[client.ts]"]
  node_viewer_backend["Viewer backend<br/>[app.py]"]
  node_viewer_index["Document indexing<br/>[indexer.py]"]
  node_viewer_document["Document loading<br/>[document.py]"]
  node_annotator["Ground-truth annotator<br/>[annotator.js]"]
end

node_user(("Benchmark user"))
node_external_parser(("Parsing service"))

node_user -->|"runs benchmark"| node_cli
node_cli -->|"downloads data"| node_dataset
node_cli -->|"selects pipeline"| node_pipeline_spec
node_cli -->|"starts inference"| node_inference_runner
node_inference_runner -->|"resolves provider"| node_provider_registry
node_inference_runner -->|"dispatches inference"| node_provider_adapters
node_provider_adapters -->|"submits documents"| node_external_parser
node_provider_adapters -->|"returns results"| node_result_models
node_inference_runner -->|"stores normalized output"| node_result_models
node_cli -->|"starts evaluation"| node_evaluation_runner
node_evaluation_runner -->|"calculates metrics"| node_metric_families
node_evaluation_runner -->|"reads cases"| node_ground_truth
node_evaluation_runner -->|"writes reports"| node_evaluation_reports
node_analysis -->|"summarizes scores"| node_evaluation_reports
node_user -->|"browses evidence"| node_viewer_frontend
node_viewer_frontend -->|"requests documents"| node_viewer_api
node_viewer_api -->|"calls API"| node_viewer_backend
node_viewer_backend -->|"indexes documents"| node_viewer_index
node_viewer_backend -->|"serves documents"| node_viewer_document
node_user -->|"reviews annotations"| node_annotator

click node_cli "https://github.com/run-llama/parsebench/blob/main/src/parse_bench/cli.py"
click node_dataset "https://github.com/run-llama/parsebench/blob/main/src/parse_bench/data/download.py"
click node_pipeline_spec "https://github.com/run-llama/parsebench/blob/main/src/parse_bench/schemas/pipeline.py"
click node_inference_runner "https://github.com/run-llama/parsebench/blob/main/src/parse_bench/inference/runner.py"
click node_provider_registry "https://github.com/run-llama/parsebench/blob/main/src/parse_bench/inference/providers/registry.py"
click node_provider_adapters "https://github.com/run-llama/parsebench/tree/main/src/parse_bench/inference/providers/parse"
click node_result_models "https://github.com/run-llama/parsebench/blob/main/src/parse_bench/schemas/pipeline_io.py"
click node_evaluation_runner "https://github.com/run-llama/parsebench/blob/main/src/parse_bench/evaluation/runner.py"
click node_metric_families "https://github.com/run-llama/parsebench/tree/main/src/parse_bench/evaluation/metrics"
click node_ground_truth "https://github.com/run-llama/parsebench/tree/main/src/parse_bench/test_cases"
click node_evaluation_reports "https://github.com/run-llama/parsebench/tree/main/src/parse_bench/evaluation/reports"
click node_analysis "https://github.com/run-llama/parsebench/tree/main/src/parse_bench/analysis"
click node_viewer_frontend "https://github.com/run-llama/parsebench/blob/main/apps/visual_grounding_viewer/frontend/src/App.tsx"
click node_viewer_api "https://github.com/run-llama/parsebench/blob/main/apps/visual_grounding_viewer/frontend/src/api/client.ts"
click node_viewer_backend "https://github.com/run-llama/parsebench/blob/main/apps/visual_grounding_viewer/backend/app.py"
click node_viewer_index "https://github.com/run-llama/parsebench/blob/main/apps/visual_grounding_viewer/backend/indexer.py"
click node_viewer_document "https://github.com/run-llama/parsebench/blob/main/apps/visual_grounding_viewer/backend/routes/document.py"
click node_annotator "https://github.com/run-llama/parsebench/blob/main/apps/annotator/annotator.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_dataset,node_user toneBlue
class node_pipeline_spec,node_inference_runner,node_provider_registry,node_provider_adapters,node_result_models toneAmber
class node_evaluation_runner,node_metric_families,node_ground_truth,node_evaluation_reports,node_analysis toneMint
class node_viewer_frontend,node_viewer_api,node_viewer_backend,node_viewer_index,node_viewer_document,node_annotator toneRose
class node_external_parser toneIndigo
```

## Components

### Benchmark workflow

- [CLI](https://github.com/run-llama/parsebench/blob/main/src/parse_bench/cli.py): `src/parse_bench/cli.py`
- [Dataset download](https://github.com/run-llama/parsebench/blob/main/src/parse_bench/data/download.py): `src/parse_bench/data/download.py`

### Inference

- [Pipeline specification](https://github.com/run-llama/parsebench/blob/main/src/parse_bench/schemas/pipeline.py): `src/parse_bench/schemas/pipeline.py`
- [Inference runner](https://github.com/run-llama/parsebench/blob/main/src/parse_bench/inference/runner.py): `src/parse_bench/inference/runner.py`
- [Provider registry](https://github.com/run-llama/parsebench/blob/main/src/parse_bench/inference/providers/registry.py): `src/parse_bench/inference/providers/registry.py`
- [Parsing providers](https://github.com/run-llama/parsebench/tree/main/src/parse_bench/inference/providers/parse): `src/parse_bench/inference/providers/parse`
- [Result models](https://github.com/run-llama/parsebench/blob/main/src/parse_bench/schemas/pipeline_io.py): `src/parse_bench/schemas/pipeline_io.py`

### Evaluation

- [Evaluation runner](https://github.com/run-llama/parsebench/blob/main/src/parse_bench/evaluation/runner.py): `src/parse_bench/evaluation/runner.py`
- [Dimension metrics](https://github.com/run-llama/parsebench/tree/main/src/parse_bench/evaluation/metrics): `src/parse_bench/evaluation/metrics`
- [Ground-truth cases](https://github.com/run-llama/parsebench/tree/main/src/parse_bench/test_cases): `src/parse_bench/test_cases`
- [Evaluation reports](https://github.com/run-llama/parsebench/tree/main/src/parse_bench/evaluation/reports): `src/parse_bench/evaluation/reports`
- [Score analysis](https://github.com/run-llama/parsebench/tree/main/src/parse_bench/analysis): `src/parse_bench/analysis`

### Grounding tools

- [Grounding viewer](https://github.com/run-llama/parsebench/blob/main/apps/visual_grounding_viewer/frontend/src/App.tsx): `apps/visual_grounding_viewer/frontend/src/App.tsx`
- [Viewer API](https://github.com/run-llama/parsebench/blob/main/apps/visual_grounding_viewer/frontend/src/api/client.ts): `apps/visual_grounding_viewer/frontend/src/api/client.ts`
- [Viewer backend](https://github.com/run-llama/parsebench/blob/main/apps/visual_grounding_viewer/backend/app.py): `apps/visual_grounding_viewer/backend/app.py`
- [Document indexing](https://github.com/run-llama/parsebench/blob/main/apps/visual_grounding_viewer/backend/indexer.py): `apps/visual_grounding_viewer/backend/indexer.py`
- [Document loading](https://github.com/run-llama/parsebench/blob/main/apps/visual_grounding_viewer/backend/routes/document.py): `apps/visual_grounding_viewer/backend/routes/document.py`
- [Ground-truth annotator](https://github.com/run-llama/parsebench/blob/main/apps/annotator/annotator.js): `apps/annotator/annotator.js`

### Other components

- **Benchmark user**
- **Parsing service**

## Connections

- Benchmark user → CLI: runs benchmark
- CLI → Dataset download: downloads data
- CLI → Pipeline specification: selects pipeline
- CLI → Inference runner: starts inference
- Inference runner → Provider registry: resolves provider
- Inference runner → Parsing providers: dispatches inference
- Parsing providers → Parsing service: submits documents
- Parsing providers → Result models: returns results
- Inference runner → Result models: stores normalized output
- CLI → Evaluation runner: starts evaluation
- Evaluation runner → Dimension metrics: calculates metrics
- Evaluation runner → Ground-truth cases: reads cases
- Evaluation runner → Evaluation reports: writes reports
- Score analysis → Evaluation reports: summarizes scores
- Benchmark user → Grounding viewer: browses evidence
- Grounding viewer → Viewer API: requests documents
- Viewer API → Viewer backend: calls API
- Viewer backend → Document indexing: indexes documents
- Viewer backend → Document loading: serves documents
- Benchmark user → Ground-truth annotator: reviews annotations