How maelkolb/passauer_jahrbuecher_pipeline works
Passauer Jahrbücher Pipeline turns a scanned volume (PDF or page-image folder) into structured, article-level digital editions and corpus data. The main workflow renders pages, applies Chandra OCR, normalizes and analyzes layout, parses the contents, locates articles and links footnotes, then emits page images, visual crops, PageXML, TEI, HTML, JSON-LD and LLM-Wiki Markdown. Configuration selects the input and OCR backend; OCR may use a local Hugging Face model or a vLLM service. The excerpts show orchestration and selected implementations; some emitter and wiki-assembly details are documented but not sampled.
Main entry points: articles.py Article boundaries · normalize.py Block normalization · wiki.py Wiki Markdown · cli.py Command line
Components 23
Input and OCR
- Command line
pjb_pipeline/cli.py - Volume config
pjb_pipeline/config.py - Page rendering
pjb_pipeline/render.py - Page images
- Chandra OCR
pjb_pipeline/ocr.py
Structure Recovery
- Block normalization
pjb_pipeline/normalize.py - Layout analysis
pjb_pipeline/structure/layout.py - Column ordering
pjb_pipeline/structure/columns.py - Contents parsing
pjb_pipeline/structure/toc.py - Person parsing
pjb_pipeline/structure/names.py - Article boundaries
pjb_pipeline/structure/articles.py - Footnote linking
pjb_pipeline/structure/footnotes.py
Edition Outputs
- Visual crops
pjb_pipeline/emit/html/crops.py - HTML edition
pjb_pipeline/emit/html/renderers.py - PageXML edition
pjb_pipeline/emit/pagexml.py - TEI edition
pjb_pipeline/emit/tei.py
Corpus Publishing
- JSON-LD graph
pjb_pipeline/emit/graph.py - JSON-LD vocabulary
pjb_pipeline/emit/jsonld_context.py - Wiki Markdown
pjb_pipeline/emit/wiki.py - Wiki assembler
pjb_pipeline/wiki_assembler.py
Other components
- Volume operator
- Scanned volume
- vLLM service
Connections 26
- Volume operator → Command line: invokes
README.md - Command line → Volume config: loads
pjb_pipeline/cli.py - Command line → Page rendering: runs pipeline
pjb_pipeline/cli.py - Volume operator → Scanned volume: provides
README.md - Page rendering → Scanned volume: reads
README.md - Page rendering → Page images: renders
- Page images → Chandra OCR: supplies pages
- Chandra OCR → vLLM service: requests OCR
README.md - Chandra OCR → Block normalization: produces blocks
pjb_pipeline/pipeline.py - Block normalization → Layout analysis: analyzes layout
pjb_pipeline/normalize.py - Layout analysis → Column ordering: uses ordering
pjb_pipeline/structure/layout.py - Block normalization → Contents parsing: provides blocks
- Contents parsing → Person parsing: resolves names
pjb_pipeline/structure/toc.py - Contents parsing → Article boundaries: defines entries
pjb_pipeline/structure/articles.py - Article boundaries → Footnote linking: organizes notes
pjb_pipeline/pipeline.py - Page images → Visual crops: provides source images
pjb_pipeline/emit/html/crops.py - Visual crops → HTML edition: embeds crops
pjb_pipeline/emit/html/renderers.py - Article boundaries → HTML edition: renders articles
- Block normalization → PageXML edition: emits pages
pjb_pipeline/pipeline.py - Article boundaries → TEI edition: emits articles
pjb_pipeline/pipeline.py - Article boundaries → JSON-LD graph: describes corpus
README.md - JSON-LD graph → JSON-LD vocabulary: uses vocabulary
pjb_pipeline/emit/graph.py - Article boundaries → Wiki Markdown: emits articles
pjb_pipeline/pipeline.py - Footnote linking → Wiki Markdown: supplies notes
pjb_pipeline/pipeline.py - Wiki Markdown → JSON-LD vocabulary: uses vocabulary
pjb_pipeline/emit/jsonld_context.py - Wiki Markdown → Wiki assembler: assembles corpus
pjb_pipeline/wiki_assembler.py