How aminediro/ferrules works
Main components
• Core Parsing Engine (ferrules-core): This is the heart of the system, containing modules for layout extraction, OCR processing, document parsing (through pdfium integration), rendering (to HTML, Markdown, JSON), block merging, and ML-driven layout inference.
• HTTP API Server (ferrules-api): Implements a RESTful interface that wraps the core engine functionality so that the document parser can be used as an online service. It listens (by default on 0.0.0.0:3002) and routes incoming requests to the core parser.
• CLI Application (ferrules-cli): Provides command-line access for quick parsing, debugging, and local usage. It leverages the core engine to parse input documents and prints/logs the output.
• External Dependencies & Libraries:
• PDF Rendering/Parsing Libraries (from the libs folder) are used for integrating pdfium functionality.
• Machine Learning Models (in the models folder) and associated Python scripts (in the python folder) that are used to convert or infer layout information.
• Additional build and deployment tools like Docker (Dockerfile, docker-compose.yml) and GitHub Actions workflows for CI/CD.
• Support/Configuration Components:
• Configuration files (e.g., Cargo.toml for each crate, .env, .cargo/config.toml) supply build and runtime parameters.
• Deployment scripts and CI/CD configurations streamline the release and testing processes.
Relationships and interactions
• Client Interaction:
• For HTTP API: Clients send PDF documents via HTTP, and the API server forwards the data to ferrules-core for processing and then returns the parsed output.
• For CLI: The command-line tool accepts file arguments, invokes ferrules-core, and outputs results (including debug visualizations).
• Data Flow:
• Input PDF documents flow into the parsing engine where pdfium (or alternative OCR tools such as Apple's Vision for OCR on macOS) extract raw text and layout.
• The parsed data is then processed through various pipelines (merging layout, block processing, and heading detection with ML acceleration).
• Finally, rendering modules format the output into various formats (HTML, Markdown, JSON).
• External Libraries and Hardware-acceleration:
Architectural patterns and design principles
• Modular Architecture and Separation of Concerns: The project cleanly separates core functionality (ferrules-core) from its user interfaces (CLI and API). These modules should be depicted as distinct components with well-defined interfaces.
• Containerization & CI/CD: Include high-level elements such as Docker and GitHub Actions workflows which are used for deployment and continuous integration testing.
• Clear labels for each component and submodule.
• Annotations for key configurable options (e.g., runtime flags, environmental variables) that influence the behavior of the system.