How unsiloed-ai/unsiloed-olmocr-benchmark works
This repository is a reproducible olmOCR-Bench harness for comparing third-party document parsers, plus a separate client for evaluating Unsiloed’s hosted parser. The primary workflow downloads benchmark PDFs, dispatches selected methods to produce scorer-layout Markdown, then invokes the unchanged upstream scorer. Local methods cover vision-language models and specialist parsing services; the API client submits PDFs, polls jobs, and writes candidate files. The README documents these capabilities, while sampled source confirms runner behavior for several providers; unsampled dispatcher wiring is represented only where documented.
Main entry points: bench.py Method dispatch · bench_via_api.py Hosted API client · gpt5/runner.py GPT-5 runner · claude_api/runner.py Claude runner
Components 20
Benchmark workflow
- Benchmark CLI
bench.py - Method dispatch
bench.py - Markdown candidates
- Comparison scores
Local runners
- GPT-5 runner
services/gpt5/runner.py - Claude runner
services/claude_api/runner.py - LlamaParse runner
services/llamaparse/runner.py - Unstructured runner
services/unstructured/runner.py - Landing AI runner
services/landing_ai/runner.py - Azure DI runner
services/azure_di/runner.py - Textract runner
services/textract/runner.py - Reducto runner
services/reducto/runner.py - Extend runner
services/extend/runner.py
Hosted API evaluation
- Hosted API client
clients/bench_via_api.py
External systems
- Benchmark dataset
- Upstream methods
- olmOCR scorer
- Unsiloed parse API
- Temporary upload storage
Other components
- Benchmark user
Connections 22
- Benchmark user → Benchmark CLI: runs commands
- Benchmark CLI → Benchmark dataset: downloads PDFs
- Benchmark CLI → Method dispatch: converts methods
- Method dispatch → Upstream methods: dispatches method
- Method dispatch → GPT-5 runner: dispatches method
- Method dispatch → Claude runner: dispatches method
- Method dispatch → LlamaParse runner: dispatches method
- Method dispatch → Unstructured runner: dispatches method
- Method dispatch → Landing AI runner: dispatches method
- Method dispatch → Azure DI runner: dispatches method
- Method dispatch → Textract runner: dispatches method
- Method dispatch → Reducto runner: dispatches method
- Method dispatch → Extend runner: dispatches method
- Method dispatch → Markdown candidates: writes Markdown
- Benchmark CLI → olmOCR scorer: runs scoring
- Markdown candidates → olmOCR scorer: scores candidates
- olmOCR scorer → Comparison scores: produces scores
- Benchmark user → Hosted API client: runs client
- Hosted API client → Benchmark dataset: reads PDFs
- Hosted API client → Unsiloed parse API: submits and polls
- Unsiloed parse API → Temporary upload storage: uploads large files
- Hosted API client → Markdown candidates: writes Markdown