# unsiloed-ai/unsiloed-olmocr-benchmark architecture

> Architecture diagram of the GitHub repository unsiloed-ai/unsiloed-olmocr-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/unsiloed-ai/unsiloed-olmocr-benchmark
- Repository: https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark

## Overview

This repository is a reproducible olmOCR-Bench harness for comparing third-party document parsers, plus a separate client for evaluating Unsiloed’s hosted parser. The primary workflow downloads benchmark PDFs, dispatches selected methods to produce scorer-layout Markdown, then invokes the unchanged upstream scorer. Local methods cover vision-language models and specialist parsing services; the API client submits PDFs, polls jobs, and writes candidate files. The README documents these capabilities, while sampled source confirms runner behavior for several providers; unsampled dispatcher wiring is represented only where documented.

Main entry points: [`bench.py`](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/bench.py) (Method dispatch), [`clients/bench_via_api.py`](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/clients/bench_via_api.py) (Hosted API client), [`services/gpt5/runner.py`](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/gpt5/runner.py) (GPT-5 runner), [`services/claude_api/runner.py`](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/claude_api/runner.py) (Claude runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Benchmark workflow"]
  node_harness["Benchmark CLI<br/>[bench.py]"]
  node_dispatch["Method dispatch<br/>[bench.py]"]
  node_candidates["Markdown candidates"]
  node_scores["Comparison scores"]
end

subgraph group_local["Local runners"]
  node_runner_gpt5["GPT-5 runner<br/>[runner.py]"]
  node_runner_claude["Claude runner<br/>[runner.py]"]
  node_runner_llama["LlamaParse runner<br/>[runner.py]"]
  node_runner_unstructured["Unstructured runner<br/>[runner.py]"]
  node_runner_landing["Landing AI runner<br/>[runner.py]"]
  node_runner_azure["Azure DI runner<br/>[runner.py]"]
  node_runner_textract["Textract runner<br/>[runner.py]"]
  node_runner_reducto["Reducto runner<br/>[runner.py]"]
  node_runner_extend["Extend runner<br/>[runner.py]"]
end

subgraph group_hosted["Hosted API evaluation"]
  node_api_client["Hosted API client<br/>[bench_via_api.py]"]
end

subgraph group_external["External systems"]
  node_dataset["Benchmark dataset"]
  node_upstream_methods["Upstream methods"]
  node_scorer["olmOCR scorer"]
  node_parse_api["Unsiloed parse API"]
  node_upload_store[("Temporary upload storage")]
end

node_engineer(("Benchmark user"))

node_engineer -->|"runs commands"| node_harness
node_harness -->|"downloads PDFs"| node_dataset
node_harness -->|"converts methods"| node_dispatch
node_dispatch -->|"dispatches method"| node_upstream_methods
node_dispatch -->|"dispatches method"| node_runner_gpt5
node_dispatch -->|"dispatches method"| node_runner_claude
node_dispatch -->|"dispatches method"| node_runner_llama
node_dispatch -->|"dispatches method"| node_runner_unstructured
node_dispatch -->|"dispatches method"| node_runner_landing
node_dispatch -->|"dispatches method"| node_runner_azure
node_dispatch -->|"dispatches method"| node_runner_textract
node_dispatch -->|"dispatches method"| node_runner_reducto
node_dispatch -->|"dispatches method"| node_runner_extend
node_dispatch -->|"writes Markdown"| node_candidates
node_harness -->|"runs scoring"| node_scorer
node_candidates -->|"scores candidates"| node_scorer
node_scorer -->|"produces scores"| node_scores
node_engineer -->|"runs client"| node_api_client
node_api_client -->|"reads PDFs"| node_dataset
node_api_client -->|"submits and polls"| node_parse_api
node_parse_api -.->|"uploads large files"| node_upload_store
node_api_client -->|"writes Markdown"| node_candidates

click node_harness "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/bench.py"
click node_dispatch "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/bench.py"
click node_runner_gpt5 "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/gpt5/runner.py"
click node_runner_claude "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/claude_api/runner.py"
click node_runner_llama "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/llamaparse/runner.py"
click node_runner_unstructured "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/unstructured/runner.py"
click node_runner_landing "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/landing_ai/runner.py"
click node_runner_azure "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/azure_di/runner.py"
click node_runner_textract "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/textract/runner.py"
click node_runner_reducto "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/reducto/runner.py"
click node_runner_extend "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/extend/runner.py"
click node_api_client "https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/clients/bench_via_api.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_harness,node_dispatch,node_candidates,node_scores,node_engineer toneBlue
class node_runner_gpt5,node_runner_claude,node_runner_llama,node_runner_unstructured,node_runner_landing,node_runner_azure,node_runner_textract,node_runner_reducto,node_runner_extend toneAmber
class node_api_client toneMint
class node_dataset,node_upstream_methods,node_scorer,node_parse_api,node_upload_store toneRose
```

## Components

### Benchmark workflow

- [Benchmark CLI](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/bench.py): `bench.py`
- [Method dispatch](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/bench.py): `bench.py`
- **Markdown candidates**
- **Comparison scores**

### Local runners

- [GPT-5 runner](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/gpt5/runner.py): `services/gpt5/runner.py`
- [Claude runner](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/claude_api/runner.py): `services/claude_api/runner.py`
- [LlamaParse runner](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/llamaparse/runner.py): `services/llamaparse/runner.py`
- [Unstructured runner](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/unstructured/runner.py): `services/unstructured/runner.py`
- [Landing AI runner](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/landing_ai/runner.py): `services/landing_ai/runner.py`
- [Azure DI runner](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/azure_di/runner.py): `services/azure_di/runner.py`
- [Textract runner](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/textract/runner.py): `services/textract/runner.py`
- [Reducto runner](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/reducto/runner.py): `services/reducto/runner.py`
- [Extend runner](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/services/extend/runner.py): `services/extend/runner.py`

### Hosted API evaluation

- [Hosted API client](https://github.com/unsiloed-ai/unsiloed-olmocr-benchmark/blob/main/clients/bench_via_api.py): `clients/bench_via_api.py`

### External systems

- **Benchmark dataset**
- **Upstream methods**
- **olmOCR scorer**
- **Unsiloed parse API**
- **Temporary upload storage**

### Other components

- **Benchmark user**

## Connections

- Benchmark user → Benchmark CLI: runs commands
- Benchmark CLI → Benchmark dataset: downloads PDFs
- Benchmark CLI → Method dispatch: converts methods
- Method dispatch → Upstream methods: dispatches method
- Method dispatch → GPT-5 runner: dispatches method
- Method dispatch → Claude runner: dispatches method
- Method dispatch → LlamaParse runner: dispatches method
- Method dispatch → Unstructured runner: dispatches method
- Method dispatch → Landing AI runner: dispatches method
- Method dispatch → Azure DI runner: dispatches method
- Method dispatch → Textract runner: dispatches method
- Method dispatch → Reducto runner: dispatches method
- Method dispatch → Extend runner: dispatches method
- Method dispatch → Markdown candidates: writes Markdown
- Benchmark CLI → olmOCR scorer: runs scoring
- Markdown candidates → olmOCR scorer: scores candidates
- olmOCR scorer → Comparison scores: produces scores
- Benchmark user → Hosted API client: runs client
- Hosted API client → Benchmark dataset: reads PDFs
- Hosted API client → Unsiloed parse API: submits and polls
- Unsiloed parse API → Temporary upload storage: uploads large files
- Hosted API client → Markdown candidates: writes Markdown
