# knaw-huc/loghi architecture

> Architecture diagram of the GitHub repository knaw-huc/loghi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/knaw-huc/loghi
- Repository: https://github.com/knaw-huc/loghi

## Overview

##### Project type and purpose
- Loghi is an open-source, end-to-end Handwritten Text Recognition (HTR) toolkit and web service.
- It provides three core processing modules (Laypa for layout analysis, Loghi-HTR for transcription, and Loghi-Tooling for pre/post-processing), script-based pipelines for CLI usage, a RESTful web service, and a Gradio demo GUI.

##### File structure
- Top-level modules:
- docker/… – Dockerfiles and build scripts for each component (laypa, htr, tooling, webservice, gradio).
- laypa/, loghi-htr/, loghi-tooling/ – Submodules for the three core ML components.
- scripts/ – Bash scripts orchestrating inference and training pipelines.
- webservice/ – REST API code, docker-compose, and helper scripts for on-demand processing.
- gradio/ – A standalone UI demo built with Gradio and its own docker-compose.
- docs/, tests/ – Documentation and integration tests.

##### Key insights from the README
- Modular pipeline: users can invoke individual stages or run the full sequence via `scripts/inference-pipeline.sh`.
- All components are containerized for ease of deployment.
- The web service wraps the same scripts behind HTTP endpoints.
- Gradio demo is an independent frontend to the same core modules.

a. Main components:
- Clients: HTTP clients (browser, CLI, curl) and the Gradio UI.
- API Layer: webservice container exposing REST endpoints.
- Pipeline Orchestrator: Bash scripts that call the three ML modules in sequence.
- ML Modules (each in its own container):
– Laypa (layout analysis via Detectron2)
– Loghi-HTR (text transcription via PyTorch)
– Loghi-Tooling (pre-/post-processing utilities)
- Model Storage: external volumes or S3-like storage for YAML configs and .pth model weights.
- Output Store: generated PageXML files, logs, synthetic data (flat files).
- Docker Registry: holds pre-built images (loghi/docker.laypa, etc.).

b. Relationships and data flow:
1. Client submits a document (image) to the Webservice API or Gradio UI.
2. The API/Gradio layer invokes the appropriate pipeline script.
3. The orchestrator script:
– Calls Laypa → produces segment masks or PageXML regions.
– Feeds regions to Loghi-HTR → produces raw text lines.
– Runs Loghi-Tooling → merges text into final PageXML and handles reading order.
4. The final PageXML is returned to the client or stored.

c. Architectural patterns and design principles:
- Modular, container-based microservices for each processing stage.
- Pipeline (chain of responsibility) pattern for the HTR workflow.
- Separation of concerns: ML logic in submodules, orchestration via scripts, and presentation via web or CLI.

d. Relevant technologies, frameworks, libraries:
- Python, Bash scripting
- Detectron2 (Laypa), PyTorch (Loghi-HTR)
- Docker (images, docker-compose)
- REST framework (Flask/FastAPI) for webservice
- Gradio for UI demo
- PageXML standard for annotation output

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Clients Layer
    subgraph "Clients" 
        direction TB
        CLI["CLI (scripts)"]:::clients
        Browser["Browser/HTTP"]:::clients
        GradioClient["Gradio UI (Browser)"]:::clients
    end

    %% Entry Points
    subgraph "Entry Points" 
        direction TB
        APIContainer["Webservice API (FastAPI/Flask)"]:::api
        GradioContainer["Gradio Demo Service"]:::ui
        CLI --> APIContainer
        Browser --> APIContainer
        GradioClient --> GradioContainer
    end

    %% Orchestration Layer
    subgraph "Orchestrator (Bash Scripts)" 
        direction TB
        ScriptsDir["scripts/"]:::orchestrator
        InferenceScript["inference-pipeline.sh"]:::orchestrator
        TrainScript["htr-train-pipeline.sh"]:::orchestrator
        ScriptsDir --> InferenceScript
        ScriptsDir --> TrainScript
        APIContainer -->|invokes| InferenceScript
        GradioContainer -->|invokes| InferenceScript
    end

    %% ML Modules
    subgraph "ML Modules (Containers)" 
        direction TB
        LaypaContainer["Laypa (Detectron2)"]:::ml
        HTRContainer["Loghi-HTR (PyTorch)"]:::ml
        ToolingContainer["Loghi-Tooling"]:::ml
        InferenceScript --> LaypaContainer
        LaypaContainer --> HTRContainer
        HTRContainer --> ToolingContainer
    end

    %% Storage Layer
    subgraph "Storage" 
        direction TB
        ModelStorage["Model Storage (S3/Volumes)"]:::storage
        OutputStore["Output Files (PageXML, logs)"]:::storage
        ToolingContainer --> ModelStorage
        ToolingContainer --> OutputStore
        APIContainer -->|returns XML| OutputStore
        GradioContainer -->|returns XML| OutputStore
    end

    %% Deployment & Registry
    subgraph "Deployment" 
        direction TB
        DockerBuild["docker/buildAll.sh"]:::deployment
        Registry["Docker Registry"]:::deployment
        DockerBuild --> Registry
        Registry --> LaypaContainer
        Registry --> HTRContainer
        Registry --> ToolingContainer
        Registry --> APIContainer
        Registry --> GradioContainer
    end

    %% Click Events
    click LaypaContainer "https://github.com/knaw-huc/loghi/blob/main/docker/docker.laypa"
    click LaypaContainer "https://github.com/knaw-huc/loghi/tree/main/laypa/"
    click HTRContainer "https://github.com/knaw-huc/loghi/blob/main/docker/docker.htr"
    click HTRContainer "https://github.com/knaw-huc/loghi/blob/main/docker/docker.htr-wsl"
    click HTRContainer "https://github.com/knaw-huc/loghi/tree/main/loghi-htr/"
    click ToolingContainer "https://github.com/knaw-huc/loghi/blob/main/docker/docker.loghi-tooling"
    click ToolingContainer "https://github.com/knaw-huc/loghi/tree/main/loghi-tooling/"
    click ScriptsDir "https://github.com/knaw-huc/loghi/tree/main/scripts/"
    click InferenceScript "https://github.com/knaw-huc/loghi/blob/main/scripts/inference-pipeline.sh"
    click TrainScript "https://github.com/knaw-huc/loghi/blob/main/scripts/htr-train-pipeline.sh"
    click APIContainer "https://github.com/knaw-huc/loghi/tree/main/webservice/"
    click APIContainer "https://github.com/knaw-huc/loghi/blob/main/webservice/docker-compose.yml"
    click GradioContainer "https://github.com/knaw-huc/loghi/tree/main/gradio/"
    click GradioContainer "https://github.com/knaw-huc/loghi/blob/main/gradio/docker/docker-compose.yml"
    click DockerBuild "https://github.com/knaw-huc/loghi/blob/main/docker/buildAll.sh"

    %% Styles
    classDef clients fill:#FFF3BF,stroke:#000
    classDef api fill:#C9E7FF,stroke:#000
    classDef ui fill:#D0F4DE,stroke:#000
    classDef orchestrator fill:#E2F0CB,stroke:#000
    classDef ml fill:#B8D8E8,stroke:#000
    classDef storage fill:#F8D7DA,stroke:#000
    classDef deployment fill:#E2E3E5,stroke:#000
```
