How knaw-huc/loghi works
Project type and purpose
• Loghi is an open-source, end-to-end Handwritten Text Recognition (HTR) toolkit and web service.
• It provides three core processing modules (Laypa for layout analysis, Loghi-HTR for transcription, and Loghi-Tooling for pre/post-processing), script-based pipelines for CLI usage, a RESTful web service, and a Gradio demo GUI.
File structure
• Top-level modules:
• docker/… – Dockerfiles and build scripts for each component (laypa, htr, tooling, webservice, gradio).
• laypa/, loghi-htr/, loghi-tooling/ – Submodules for the three core ML components.
• scripts/ – Bash scripts orchestrating inference and training pipelines.
• webservice/ – REST API code, docker-compose, and helper scripts for on-demand processing.
• gradio/ – A standalone UI demo built with Gradio and its own docker-compose.
• docs/, tests/ – Documentation and integration tests.
Key insights from the README
• Modular pipeline: users can invoke individual stages or run the full sequence via scripts/inference-pipeline.sh.
• All components are containerized for ease of deployment.
• The web service wraps the same scripts behind HTTP endpoints.
• Gradio demo is an independent frontend to the same core modules.
a. Main components:
• Clients: HTTP clients (browser, CLI, curl) and the Gradio UI.
• API Layer: webservice container exposing REST endpoints.
• Pipeline Orchestrator: Bash scripts that call the three ML modules in sequence.
• ML Modules (each in its own container):
– Laypa (layout analysis via Detectron2)
– Loghi-HTR (text transcription via PyTorch)
– Loghi-Tooling (pre-/post-processing utilities)
• Model Storage: external volumes or S3-like storage for YAML configs and .pth model weights.
• Output Store: generated PageXML files, logs, synthetic data (flat files).
• Docker Registry: holds pre-built images (loghi/docker.laypa, etc.).
b. Relationships and data flow:
1. Client submits a document (image) to the Webservice API or Gradio UI.
2. The API/Gradio layer invokes the appropriate pipeline script.
3. The orchestrator script:
– Calls Laypa → produces segment masks or PageXML regions.
– Feeds regions to Loghi-HTR → produces raw text lines.
– Runs Loghi-Tooling → merges text into final PageXML and handles reading order.
4. The final PageXML is returned to the client or stored.
c. Architectural patterns and design principles:
• Modular, container-based microservices for each processing stage.
• Pipeline (chain of responsibility) pattern for the HTR workflow.
• Separation of concerns: ML logic in submodules, orchestration via scripts, and presentation via web or CLI.
d. Relevant technologies, frameworks, libraries:
• Python, Bash scripting
• Detectron2 (Laypa), PyTorch (Loghi-HTR)
• Docker (images, docker-compose)
• REST framework (Flask/FastAPI) for webservice
• Gradio for UI demo
• PageXML standard for annotation output