# aminediro/ferrules architecture

> Architecture diagram of the GitHub repository aminediro/ferrules, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aminediro/ferrules
- Repository: https://github.com/aminediro/ferrules

## Overview

##### Main components
- Core Parsing Engine (ferrules-core): This is the heart of the system, containing modules for layout extraction, OCR processing, document parsing (through pdfium integration), rendering (to HTML, Markdown, JSON), block merging, and ML-driven layout inference.
- HTTP API Server (ferrules-api): Implements a RESTful interface that wraps the core engine functionality so that the document parser can be used as an online service. It listens (by default on 0.0.0.0:3002) and routes incoming requests to the core parser.
- CLI Application (ferrules-cli): Provides command-line access for quick parsing, debugging, and local usage. It leverages the core engine to parse input documents and prints/logs the output.
- External Dependencies & Libraries:
- PDF Rendering/Parsing Libraries (from the libs folder) are used for integrating pdfium functionality.
- Machine Learning Models (in the models folder) and associated Python scripts (in the python folder) that are used to convert or infer layout information.
- Additional build and deployment tools like Docker (Dockerfile, docker-compose.yml) and GitHub Actions workflows for CI/CD.
- Support/Configuration Components:
- Configuration files (e.g., Cargo.toml for each crate, .env, .cargo/config.toml) supply build and runtime parameters.
- Deployment scripts and CI/CD configurations streamline the release and testing processes.

##### Relationships and interactions
- Client Interaction:
- For HTTP API: Clients send PDF documents via HTTP, and the API server forwards the data to ferrules-core for processing and then returns the parsed output.
- For CLI: The command-line tool accepts file arguments, invokes ferrules-core, and outputs results (including debug visualizations).
- Data Flow:
- Input PDF documents flow into the parsing engine where pdfium (or alternative OCR tools such as Apple's Vision for OCR on macOS) extract raw text and layout.
- The parsed data is then processed through various pipelines (merging layout, block processing, and heading detection with ML acceleration).
- Finally, rendering modules format the output into various formats (HTML, Markdown, JSON).
- External Libraries and Hardware-acceleration:

##### Architectural patterns and design principles
- Modular Architecture and Separation of Concerns: The project cleanly separates core functionality (ferrules-core) from its user interfaces (CLI and API). These modules should be depicted as distinct components with well-defined interfaces.
- Containerization & CI/CD: Include high-level elements such as Docker and GitHub Actions workflows which are used for deployment and continuous integration testing.

- Clear labels for each component and submodule.
- Annotations for key configurable options (e.g., runtime flags, environmental variables) that influence the behavior of the system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Client Layer
    U["User/Client"]:::client

    %% Interface Layers
    API["HTTP API Server (ferrules-api)"]:::interface
    CLI["CLI Application (ferrules-cli)"]:::interface

    %% Input Node
    Input["Input PDF Document"]:::input

    %% Ferrules Core Subgraph
    subgraph "Ferrules Core (ferrules-core)"
        CR1["OCR Processing"]:::core
        CR2["Document Parsing (pdfium)"]:::core
        CR3["Layout Extraction"]:::core
        CR4["Block Merging"]:::core
        CR5["ML-driven Layout Inference"]:::core
        CR6["Rendering (HTML/Markdown/JSON)"]:::core
    end

    %% External Dependencies
    PDFium["PDFium Libraries"]:::external
    ML["ML/Inference Engines"]:::external
    Models["ML Models"]:::external
    Python["Python Scripts for ML/Inference"]:::external

    %% Support Components
    Config["Configuration Files"]:::config
    Deploy["Deployment Tools (Docker & CI/CD)"]:::deployment

    %% Client to Interfaces
    U -->|"sends request"| API
    U -->|"uses CLI"| CLI

    %% Interfaces to Input
    API -->|"uploads PDF"| Input
    CLI -->|"provides file"| Input

    %% Input to Ferrules Core Pipeline (Parallel ingestion)
    Input --> CR1
    Input --> CR2

    %% Internal Ferrules Core Processing Flow
    CR1 --> CR3
    CR2 --> CR3
    CR3 --> CR4
    CR4 --> CR5
    CR5 --> CR6
    CR6 -->|"returns output"| U

    %% External Dependencies Integration
    PDFium -->|"supports parsing"| CR2
    ML -->|"accelerates inference"| CR5
    Models -->|"feeds ML inference"| CR5
    Python -->|"manages model conversion"| Models

    %% Support/Configuration Integration
    Config -->|"configures"| API
    Config -->|"configures"| CLI
    Config -->|"configures"| CR1
    Deploy -->|"deploys"| API
    Deploy -->|"deploys"| CLI
    Deploy -->|"deploys"| CR1

    %% Click Events
    click API "https://github.com/aminediro/ferrules/tree/main/ferrules-api/"
    click CLI "https://github.com/aminediro/ferrules/tree/main/ferrules-cli/"
    click CR1 "https://github.com/aminediro/ferrules/tree/main/ferrules-core/"
    click PDFium "https://github.com/aminediro/ferrules/tree/main/libs/"
    click Models "https://github.com/aminediro/ferrules/blob/main/models/yolov8s-doclaynet.onnx"
    click Python "https://github.com/aminediro/ferrules/blob/main/python/infer.py"
    click Deploy "https://github.com/aminediro/ferrules/tree/main/Dockerfile"
    click Config "https://github.com/aminediro/ferrules/blob/main/ferrules-core/Cargo.toml"

    %% Styles
    classDef client fill:#f9e79f,stroke:#b7950b,stroke-width:2px;
    classDef interface fill:#aed6f1,stroke:#2471a3,stroke-width:2px;
    classDef core fill:#abebc6,stroke:#1e8449,stroke-width:2px;
    classDef external fill:#f5cba7,stroke:#d35400,stroke-width:2px;
    classDef deployment fill:#d2b4de,stroke:#6c3483,stroke-width:2px;
    classDef config fill:#f9c2ff,stroke:#a569bd,stroke-width:2px;
    classDef input fill:#fad7a0,stroke:#d35400,stroke-width:2px;
```
