# maelkolb/passauer_jahrbuecher_pipeline architecture

> Architecture diagram of the GitHub repository maelkolb/passauer_jahrbuecher_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/maelkolb/passauer_jahrbuecher_pipeline
- Repository: https://github.com/maelkolb/passauer_jahrbuecher_pipeline

## Overview

Passauer Jahrbücher Pipeline turns a scanned volume (PDF or page-image folder) into structured, article-level digital editions and corpus data. The main workflow renders pages, applies Chandra OCR, normalizes and analyzes layout, parses the contents, locates articles and links footnotes, then emits page images, visual crops, PageXML, TEI, HTML, JSON-LD and LLM-Wiki Markdown. Configuration selects the input and OCR backend; OCR may use a local Hugging Face model or a vLLM service. The excerpts show orchestration and selected implementations; some emitter and wiki-assembly details are documented but not sampled.

Main entry points: [`pjb_pipeline/structure/articles.py`](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/articles.py) (Article boundaries), [`pjb_pipeline/normalize.py`](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/normalize.py) (Block normalization), [`pjb_pipeline/emit/wiki.py`](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/wiki.py) (Wiki Markdown), [`pjb_pipeline/cli.py`](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/cli.py) (Command line)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Input and OCR"]
  node_cli["Command line<br/>[cli.py]"]
  node_config["Volume config<br/>[config.py]"]
  node_render["Page rendering<br/>[render.py]"]
  node_pages["Page images"]
  node_ocr["Chandra OCR<br/>[ocr.py]"]
end

subgraph group_structure["Structure Recovery"]
  node_normalize["Block normalization<br/>[normalize.py]"]
  node_layout["Layout analysis<br/>[layout.py]"]
  node_columns["Column ordering<br/>[columns.py]"]
  node_toc["Contents parsing<br/>[toc.py]"]
  node_names["Person parsing<br/>[names.py]"]
  node_articles["Article boundaries<br/>[articles.py]"]
  node_footnotes["Footnote linking<br/>[footnotes.py]"]
end

subgraph group_edition["Edition Outputs"]
  node_regions["Visual crops<br/>[crops.py]"]
  node_html["HTML edition<br/>[renderers.py]"]
  node_pagexml["PageXML edition<br/>[pagexml.py]"]
  node_tei["TEI edition<br/>[tei.py]"]
end

subgraph group_corpus["Corpus Publishing"]
  node_graph["JSON-LD graph<br/>[graph.py]"]
  node_context["JSON-LD vocabulary<br/>[jsonld_context.py]"]
  node_wiki["Wiki Markdown<br/>[wiki.py]"]
  node_assembler["Wiki assembler<br/>[wiki_assembler.py]"]
end

node_operator(("Volume operator"))
node_scans["Scanned volume"]
node_ocrservice["vLLM service"]

node_operator -->|"invokes"| node_cli
node_cli -->|"loads"| node_config
node_cli -->|"runs pipeline"| node_render
node_operator -->|"provides"| node_scans
node_render -->|"reads"| node_scans
node_render -->|"renders"| node_pages
node_pages -->|"supplies pages"| node_ocr
node_ocr -.->|"requests OCR"| node_ocrservice
node_ocr -->|"produces blocks"| node_normalize
node_normalize -->|"analyzes layout"| node_layout
node_layout -->|"uses ordering"| node_columns
node_normalize -.->|"provides blocks"| node_toc
node_toc -->|"resolves names"| node_names
node_toc -->|"defines entries"| node_articles
node_articles -->|"organizes notes"| node_footnotes
node_pages -->|"provides source images"| node_regions
node_regions -->|"embeds crops"| node_html
node_articles -.->|"renders articles"| node_html
node_normalize -->|"emits pages"| node_pagexml
node_articles -->|"emits articles"| node_tei
node_articles -.->|"describes corpus"| node_graph
node_graph -->|"uses vocabulary"| node_context
node_articles -->|"emits articles"| node_wiki
node_footnotes -->|"supplies notes"| node_wiki
node_wiki -->|"uses vocabulary"| node_context
node_wiki -->|"assembles corpus"| node_assembler

click node_cli "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/cli.py"
click node_config "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/config.py"
click node_render "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/render.py"
click node_ocr "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/ocr.py"
click node_normalize "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/normalize.py"
click node_layout "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/layout.py"
click node_columns "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/columns.py"
click node_toc "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/toc.py"
click node_names "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/names.py"
click node_articles "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/articles.py"
click node_footnotes "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/footnotes.py"
click node_regions "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/html/crops.py"
click node_html "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/html/renderers.py"
click node_pagexml "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/pagexml.py"
click node_tei "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/tei.py"
click node_graph "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/graph.py"
click node_context "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/jsonld_context.py"
click node_wiki "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/wiki.py"
click node_assembler "https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/wiki_assembler.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config,node_render,node_pages,node_ocr toneBlue
class node_normalize,node_layout,node_columns,node_toc,node_names,node_articles,node_footnotes toneAmber
class node_regions,node_html,node_pagexml,node_tei toneMint
class node_graph,node_context,node_wiki,node_assembler toneRose
class node_operator,node_scans,node_ocrservice toneIndigo
```

## Components

### Input and OCR

- [Command line](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/cli.py): `pjb_pipeline/cli.py`
- [Volume config](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/config.py): `pjb_pipeline/config.py`
- [Page rendering](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/render.py): `pjb_pipeline/render.py`
- **Page images**
- [Chandra OCR](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/ocr.py): `pjb_pipeline/ocr.py`

### Structure Recovery

- [Block normalization](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/normalize.py): `pjb_pipeline/normalize.py`
- [Layout analysis](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/layout.py): `pjb_pipeline/structure/layout.py`
- [Column ordering](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/columns.py): `pjb_pipeline/structure/columns.py`
- [Contents parsing](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/toc.py): `pjb_pipeline/structure/toc.py`
- [Person parsing](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/names.py): `pjb_pipeline/structure/names.py`
- [Article boundaries](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/articles.py): `pjb_pipeline/structure/articles.py`
- [Footnote linking](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/structure/footnotes.py): `pjb_pipeline/structure/footnotes.py`

### Edition Outputs

- [Visual crops](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/html/crops.py): `pjb_pipeline/emit/html/crops.py`
- [HTML edition](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/html/renderers.py): `pjb_pipeline/emit/html/renderers.py`
- [PageXML edition](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/pagexml.py): `pjb_pipeline/emit/pagexml.py`
- [TEI edition](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/tei.py): `pjb_pipeline/emit/tei.py`

### Corpus Publishing

- [JSON-LD graph](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/graph.py): `pjb_pipeline/emit/graph.py`
- [JSON-LD vocabulary](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/jsonld_context.py): `pjb_pipeline/emit/jsonld_context.py`
- [Wiki Markdown](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/emit/wiki.py): `pjb_pipeline/emit/wiki.py`
- [Wiki assembler](https://github.com/maelkolb/passauer_jahrbuecher_pipeline/blob/main/pjb_pipeline/wiki_assembler.py): `pjb_pipeline/wiki_assembler.py`

### Other components

- **Volume operator**
- **Scanned volume**
- **vLLM service**

## Connections

- Volume operator → Command line: invokes (evidence: `README.md`)
- Command line → Volume config: loads (evidence: `pjb_pipeline/cli.py`)
- Command line → Page rendering: runs pipeline (evidence: `pjb_pipeline/cli.py`)
- Volume operator → Scanned volume: provides (evidence: `README.md`)
- Page rendering → Scanned volume: reads (evidence: `README.md`)
- Page rendering → Page images: renders
- Page images → Chandra OCR: supplies pages
- Chandra OCR → vLLM service: requests OCR (evidence: `README.md`)
- Chandra OCR → Block normalization: produces blocks (evidence: `pjb_pipeline/pipeline.py`)
- Block normalization → Layout analysis: analyzes layout (evidence: `pjb_pipeline/normalize.py`)
- Layout analysis → Column ordering: uses ordering (evidence: `pjb_pipeline/structure/layout.py`)
- Block normalization → Contents parsing: provides blocks
- Contents parsing → Person parsing: resolves names (evidence: `pjb_pipeline/structure/toc.py`)
- Contents parsing → Article boundaries: defines entries (evidence: `pjb_pipeline/structure/articles.py`)
- Article boundaries → Footnote linking: organizes notes (evidence: `pjb_pipeline/pipeline.py`)
- Page images → Visual crops: provides source images (evidence: `pjb_pipeline/emit/html/crops.py`)
- Visual crops → HTML edition: embeds crops (evidence: `pjb_pipeline/emit/html/renderers.py`)
- Article boundaries → HTML edition: renders articles
- Block normalization → PageXML edition: emits pages (evidence: `pjb_pipeline/pipeline.py`)
- Article boundaries → TEI edition: emits articles (evidence: `pjb_pipeline/pipeline.py`)
- Article boundaries → JSON-LD graph: describes corpus (evidence: `README.md`)
- JSON-LD graph → JSON-LD vocabulary: uses vocabulary (evidence: `pjb_pipeline/emit/graph.py`)
- Article boundaries → Wiki Markdown: emits articles (evidence: `pjb_pipeline/pipeline.py`)
- Footnote linking → Wiki Markdown: supplies notes (evidence: `pjb_pipeline/pipeline.py`)
- Wiki Markdown → JSON-LD vocabulary: uses vocabulary (evidence: `pjb_pipeline/emit/jsonld_context.py`)
- Wiki Markdown → Wiki assembler: assembles corpus (evidence: `pjb_pipeline/wiki_assembler.py`)
