# quivrhq/megaparse architecture

> Architecture diagram of the GitHub repository quivrhq/megaparse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/quivrhq/megaparse
- Repository: https://github.com/quivrhq/megaparse

## Overview

##### Main Components
- Core Parsing Engine – Represented by the code under libs/megaparse. This includes:
- Parsing modules: Multiple parsers (e.g., doctr_parser, unstructured_parser, llama, megparse_vision) driving the document-to-text extraction.
- Layout Detection: Using ONNX models (e.g., YOLO-based detection via the layout_detection folder) to handle image/PDF processing.
- Formatter Components: Such as structured_formatter and table_formatter for rendering output data.
- Exception and configuration management subsystems.
- SDK/Client Library – Located in libs/megaparse_sdk. It provides an abstraction layer, client configuration, and endpoints for external integration.
- API Service – Indicated by the README example that shows API usage (Makefile targets invoke a backend service) and the docker-compose/UAPI docs served at localhost:8000/docs. This service wraps the core parsing engine to provide REST endpoints.
- External Dependencies & Tools – These include language model integrations (OpenAI, Anthropic), as well as prerequisite tools like poppler (for PDF processing), tesseract (for OCR), and potentially libmagic (on macOS environments).
- DevOps and Deployment: Multiple Dockerfiles (normal and GPU enabled), docker-compose files, AWS task definition file (.aws/task_definition.json), and GitHub workflows which indicate continuous integration, testing, and deployment pipelines.

##### Relationships and Interactions
- Input Flow: The system starts with document submissions, either via the API endpoints or via the SDK’s file and URL upload methods.
- Ingestion Process: The document is first processed by file-specific extraction modules. The layout detection module runs (e.g., via onnx and YOLO models) for documents that require imaging detection.
- Parsing Pipeline: Depending on the type of document and configured parser (among several provided modules), the document is parsed. Errors are handled through the exceptions subsystem.
- Formatting and Postprocessing: After parsing, the formatter (structured or table based) converts the raw output into a clean, user-readable, or downstream-consumable format.
- External API Calls: Some modules (for instance, those invoking ChatOpenAI or other language models) may send data over to external services and integrate their responses.
- Output Flow: The processed/parsed document is returned to the client via the API response or SDK callback.
- Supporting Services: Environment configuration (.env), testing (benchmark and evaluations folders), and continuous integration (GitHub/workflows) serve as crucial supporting components.

##### Architectural Patterns and Design Principles
- Modular design: The project is separated into a core parsing/processing library and an SDK layer. This modularity allows easy integration with external applications.
- Layered Architecture: With distinct layers (document ingestion, parsing, formatting, API exposure), the separation of concerns is clear.
- Extensibility: The existence of multiple parser implementations and formatter modules indicates that new parsing strategies or postprocessing modules can be integrated with minimal friction.
- Containerization & DevOps Integration: The Dockerfiles, docker-compose, and AWS definitions allow deployment in various environments (including GPU-enabled ones) and integration with CI/CD pipelines.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        Clients["External Clients/Users"]:::client
        API_Service["API Service"]:::client
        SDK_Client["SDK Client"]:::client
    end

    Clients --> API_Service
    Clients --> SDK_Client

    %% SDK Endpoints
    subgraph "SDK Endpoints"
        FileUpload["File Upload Endpoint"]:::client
        URLUpload["URL Upload Endpoint"]:::client
        Schema["SDK Schema"]:::client
    end

    SDK_Client -->|"uses"| FileUpload
    SDK_Client -->|"uses"| URLUpload
    SDK_Client -->|"validates"| Schema

    %% Core Parsing Engine
    subgraph "Core Parsing Engine"
        Ingestion["Document Ingestion"]:::core
        Layout["Layout Detection"]:::core
        Parsers["Parsing Modules"]:::core
        
        %% Parser Modules Subgraph
        subgraph "Parser Modules"
            Doctr["Doctr Parser"]:::core
            Unstructured["Unstructured Parser"]:::core
            Llama["Llama Parser"]:::core
            Vision["MegaParse Vision"]:::core
        end

        %% Formatter Modules Subgraph
        subgraph "Formatter Modules"
            Structured["Structured Formatter"]:::core
            TableLLM["LLM Table Formatter"]:::core
            TableVision["Vision Table Formatter"]:::core
        end

        %% Exception and Configuration Management Subgraph
        subgraph "Exception & Config"
            ExceptionHandler["Megaparse Exceptions"]:::core
            BaseExceptions["Base Exceptions"]:::core
            Configuration["Configuration Management"]:::core
        end
    end

    %% Data Flow within Core Parsing Engine
    API_Service -->|"ingest"| Ingestion
    SDK_Client -->|"ingest"| Ingestion

    Ingestion -->|"preprocess"| Layout
    Ingestion -->|"parse"| Parsers
    Layout -->|"enhance"| Parsers

    Parsers -->|"format"| Structured
    Parsers -->|"format"| TableLLM
    Parsers -->|"format"| TableVision

    Parsers -->|"error"| ExceptionHandler
    Parsers -->|"error_base"| BaseExceptions
    Configuration --- Parsers

    %% Output Flow
    Structured -->|"output"| API_Service
    TableLLM -->|"output"| API_Service
    TableVision -->|"output"| API_Service

    %% External Integrations
    subgraph "External Integrations"
        LLM_Integration["Language Model Integration"]:::external
        DocTools["Document Processing Tools"]:::external
    end

    Parsers -->|"LM_call"| LLM_Integration
    Layout -->|"tool_call"| DocTools

    %% DevOps & Deployment
    subgraph "DevOps & Deployment"
        subgraph "Docker"
            Dockerfile["Dockerfile"]:::devops
            Dockerfile_GPU["Dockerfile.gpu"]:::devops
        end
        subgraph "Compose"
            ComposeFile["docker-compose.yml"]:::devops
            ComposeDev["docker-compose.dev.yml"]:::devops
        end
        AWS["AWS Deployment"]:::devops
        subgraph "CI/CD"
            Github_CI1["CI.yml"]:::devops
            Github_CI2["build-and-deploy.yml"]:::devops
            Github_CI3["build-gpu.yml"]:::devops
            Github_CI4["release-please.yml"]:::devops
            Github_CI5["test-build-docker.yml"]:::devops
        end
        Env[".env.example"]:::devops
    end

    %% Indicate Deployment Connections
    API_Service ---|"deployed_via"| Dockerfile
    API_Service ---|"deployed_via"| ComposeFile
    API_Service ---|"deployed_via"| AWS
    API_Service ---|"deployed_via"| Github_CI1
    API_Service ---|"deployed_via"| Env

    %% Styles
    classDef client fill:#ffcc00,stroke:#333,stroke-width:2px;
    classDef core fill:#66ccff,stroke:#333,stroke-width:2px;
    classDef external fill:#a0d468,stroke:#333,stroke-width:2px;
    classDef devops fill:#ff9966,stroke:#333,stroke-width:2px;

    %% Click Events for Core Parsing Engine
    click Doctr "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse/src/megaparse/parser/doctr_parser.py"
    click Unstructured "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse/src/megaparse/parser/unstructured_parser.py"
    click Llama "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse/src/megaparse/parser/llama.py"
    click Vision "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse/src/megaparse/parser/megaparse_vision.py"
    click Layout "https://github.com/quivrhq/megaparse/tree/main/libs/megaparse/src/megaparse/layout_detection"
    click Structured "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse/src/megaparse/formatter/structured_formatter/custom_structured_formatter.py"
    click TableLLM "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse/src/megaparse/formatter/table_formatter/llm_table_formatter.py"
    click TableVision "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse/src/megaparse/formatter/table_formatter/vision_table_formatter.py"
    click ExceptionHandler "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse/src/megaparse/api/exceptions/megaparse_exceptions.py"
    click BaseExceptions "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse/src/megaparse/exceptions/base.py"
    click Configuration "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse/src/megaparse/configs/auto.py"

    %% Click Events for SDK/Client Library
    click SDK_Client "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse_sdk/megaparse_sdk/client.py"
    click FileUpload "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse_sdk/megaparse_sdk/endpoints/file_upload.py"
    click URLUpload "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse_sdk/megaparse_sdk/endpoints/url_upload.py"
    click Schema "https://github.com/quivrhq/megaparse/tree/main/libs/megaparse_sdk/megaparse_sdk/schema/"

    %% Click Event for API Service
    click API_Service "https://github.com/quivrhq/megaparse/blob/main/libs/megaparse/src/megaparse/api/app.py"

    %% Click Events for DevOps & Deployment
    click Dockerfile "https://github.com/quivrhq/megaparse/tree/main/Dockerfile"
    click Dockerfile_GPU "https://github.com/quivrhq/megaparse/blob/main/Dockerfile.gpu"
    click ComposeFile "https://github.com/quivrhq/megaparse/blob/main/docker-compose.yml"
    click ComposeDev "https://github.com/quivrhq/megaparse/blob/main/docker-compose.dev.yml"
    click AWS "https://github.com/quivrhq/megaparse/blob/main/ .aws/task_definition.json"
    click Github_CI1 "https://github.com/quivrhq/megaparse/blob/main/.github/workflows/CI.yml"
    click Github_CI2 "https://github.com/quivrhq/megaparse/blob/main/.github/workflows/build-and-deploy.yml"
    click Github_CI3 "https://github.com/quivrhq/megaparse/blob/main/.github/workflows/build-gpu.yml"
    click Github_CI4 "https://github.com/quivrhq/megaparse/blob/main/.github/workflows/release-please.yml"
    click Github_CI5 "https://github.com/quivrhq/megaparse/blob/main/.github/workflows/test-build-docker.yml"
    click Env "https://github.com/quivrhq/megaparse/blob/main/.env.example"
```
