How run-llama/parsebench works
ParseBench is a document-parsing benchmark: users select a pipeline, run document inference against a parsing provider, evaluate normalized results against the benchmark’s ground truth, and inspect reports or leaderboard comparisons. Its five documented dimensions—tables, charts, content faithfulness, semantic formatting, and visual grounding—are represented in evaluation. The repository also includes a visual-grounding browser and an annotator for inspecting or editing page-level evidence. The README establishes the benchmark workflow; only selected runtime sources were sampled, so uncertain internal wiring is omitted.
Main entry points: cli.py CLI · inference/runner.py Inference runner · evaluation/runner.py Evaluation runner · parse Parsing providers
Components 20
Benchmark workflow
- CLI
src/parse_bench/cli.py - Dataset download
src/parse_bench/data/download.py
Inference
- Pipeline specification
src/parse_bench/schemas/pipeline.py - Inference runner
src/parse_bench/inference/runner.py - Provider registry
src/parse_bench/inference/providers/registry.py - Parsing providers
src/parse_bench/inference/providers/parse - Result models
src/parse_bench/schemas/pipeline_io.py
Evaluation
- Evaluation runner
src/parse_bench/evaluation/runner.py - Dimension metrics
src/parse_bench/evaluation/metrics - Ground-truth cases
src/parse_bench/test_cases - Evaluation reports
src/parse_bench/evaluation/reports - Score analysis
src/parse_bench/analysis
Grounding tools
- Grounding viewer
apps/visual_grounding_viewer/frontend/src/App.tsx - Viewer API
apps/visual_grounding_viewer/frontend/src/api/client.ts - Viewer backend
apps/visual_grounding_viewer/backend/app.py - Document indexing
apps/visual_grounding_viewer/backend/indexer.py - Document loading
apps/visual_grounding_viewer/backend/routes/document.py - Ground-truth annotator
apps/annotator/annotator.js
Other components
- Benchmark user
- Parsing service
Connections 20
- Benchmark user → CLI: runs benchmark
- CLI → Dataset download: downloads data
- CLI → Pipeline specification: selects pipeline
- CLI → Inference runner: starts inference
- Inference runner → Provider registry: resolves provider
- Inference runner → Parsing providers: dispatches inference
- Parsing providers → Parsing service: submits documents
- Parsing providers → Result models: returns results
- Inference runner → Result models: stores normalized output
- CLI → Evaluation runner: starts evaluation
- Evaluation runner → Dimension metrics: calculates metrics
- Evaluation runner → Ground-truth cases: reads cases
- Evaluation runner → Evaluation reports: writes reports
- Score analysis → Evaluation reports: summarizes scores
- Benchmark user → Grounding viewer: browses evidence
- Grounding viewer → Viewer API: requests documents
- Viewer API → Viewer backend: calls API
- Viewer backend → Document indexing: indexes documents
- Viewer backend → Document loading: serves documents
- Benchmark user → Ground-truth annotator: reviews annotations