# nanonets/docext architecture

> Architecture diagram of the GitHub repository nanonets/docext, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nanonets/docext
- Repository: https://github.com/nanonets/docext

## Overview

##### Project type and purpose
- docext is an open-source, on-premises toolkit for document information extraction and benchmarking.
- It provides:
- A command-line/UI client and REST API for users to submit documents.
- A core extraction library that leverages vision-language models (VLMs) to extract fields, tables, OCR text, etc.
- A benchmarking suite to evaluate VLMs on various document intelligence tasks (KIE, VQA, OCR, tables, classification, confidence calibration, long-document reasoning).

##### File structure
- Top level:
- Dockerfile & configs/benchmark.yaml for deployment & configuration
- requirement.txt, setup.py for packaging
- EXT_README.md, Troubleshooting.md, CONTRIBUTING docs, .github workflows
- Package “docext”:
- app/ → Application layer: CLI/UI (args.py, ui.py), REST API server (app.py), utility wrappers (utils.py)
- core/ → Extraction engine: client.py (model client abstraction), extract.py (orchestration), prompts.py, confidence.py, config.py
- benchmark/ → Benchmarking suite: tasks.py (benchmark orchestration), metrics/ (various metric calculators), vlm_datasets/ (dataset loaders), utils.py

##### README for additional insights
- The toolkit runs entirely on-premises, supports multi-page docs, customizable templates, REST API endpoints, and a live leaderboard.
- Dependencies include Python, Docker, vision-language model backends (via core/vllm.py).

## Diagram (Mermaid)

```mermaid
graph TB
    %% User
    User["User"]:::external

    %% UI Layer
    subgraph "User Interfaces"
        CLIArgs["CLI Args Parser"]:::frontend
        UI["CLI/UI Layer"]:::frontend
        API["REST API Server"]:::frontend
        UtilsApp["App Utilities"]:::frontend
    end

    %% Core Extraction Service
    subgraph "Core Extraction Service"
        Config["Configuration Manager"]:::core
        Extract["Inference Orchestrator"]:::core
        Prompts["Prompt Manager"]:::core
        Client["Model Client"]:::core
        VLLM["vLLM Backend Adapter"]:::core
        Confidence["Confidence Scorer"]:::core
        UtilsCore["Core Utilities"]:::core
    end

    %% Benchmarking Module
    subgraph "Benchmarking Module"
        BenchEntry["Benchmark Entry Point"]:::bench
        TaskRunner["Benchmark Task Runner"]:::bench
        Dataset["Dataset Loaders"]:::bench
        MetricsClassification["Classification Metrics"]:::bench
        MetricsKIE["KIE Metrics"]:::bench
        MetricsOCR["OCR Metrics"]:::bench
        MetricsTables["Tables Metrics"]:::bench
        MetricsVQA["VQA Metrics"]:::bench
        UtilsBench["Benchmark Utilities"]:::bench
    end

    %% External Services & Resources
    subgraph "External Services"
        VLMBackend["Vision-Language Model Backends"]:::external
        Storage["Data Storage (optional)"]:::external
    end

    %% Deployment & Configuration
    subgraph "Deployment & Configuration"
        Docker["Docker Container"]:::config
        ConfigFiles["YAML Configs"]:::config
        Setup["Packaging & Setup"]:::config
    end

    %% Flow: User to UI/API
    User -->|"via CLI"| CLIArgs
    User -->|"via CLI/UI"| UI
    User -->|"via REST API"| API
    UI -->|invokes| UtilsApp
    API -->|invokes| UtilsApp

    %% UI/API to Core
    CLIArgs -->|submit document| Extract
    UI -->|submit document| Extract
    API -->|REST request| Extract

    %% Core internal interactions
    Extract -->|load configs| Config
    Extract -->|use prompts| Prompts
    Extract -->|invoke model client| Client
    Client -->|adapter| VLLM
    VLMBackend -->|responses| Client
    Extract -->|score confidence| Confidence
    Extract -->|util functions| UtilsCore
    Confidence -->|util functions| UtilsCore

    %% Response back to UI/API
    Extract -->|JSON response| UI
    Extract -->|JSON response| API

    %% Benchmark flow
    BenchEntry -->|start benchmark| TaskRunner
    TaskRunner -->|load data| Dataset
    TaskRunner -->|call core extract| Extract
    Extract -->|result| TaskRunner
    TaskRunner -->|compute| MetricsClassification
    TaskRunner -->|compute| MetricsKIE
    TaskRunner -->|compute| MetricsOCR
    TaskRunner -->|compute| MetricsTables
    TaskRunner -->|compute| MetricsVQA
    TaskRunner -->|utils| UtilsBench
    TaskRunner -->|store results| Storage

    %% Configuration influences
    ConfigFiles -->|drives| Extract
    ConfigFiles -->|drives| TaskRunner

    %% Packaging & Deployment
    Docker -->|contains| CLIArgs
    Docker -->|contains| API
    Docker -->|contains| Extract
    Docker -->|contains| BenchEntry
    Setup -->|defines deps| Docker

    %% Click Events
    click CLIArgs "https://github.com/nanonets/docext/blob/main/docext/app/args.py"
    click UI "https://github.com/nanonets/docext/blob/main/docext/app/ui.py"
    click API "https://github.com/nanonets/docext/blob/main/docext/app/app.py"
    click UtilsApp "https://github.com/nanonets/docext/blob/main/docext/app/utils.py"
    click Config "https://github.com/nanonets/docext/blob/main/docext/core/config.py"
    click Extract "https://github.com/nanonets/docext/blob/main/docext/core/extract.py"
    click Client "https://github.com/nanonets/docext/blob/main/docext/core/client.py"
    click VLLM "https://github.com/nanonets/docext/blob/main/docext/core/vllm.py"
    click Prompts "https://github.com/nanonets/docext/blob/main/docext/core/prompts.py"
    click Confidence "https://github.com/nanonets/docext/blob/main/docext/core/confidence.py"
    click UtilsCore "https://github.com/nanonets/docext/blob/main/docext/core/utils.py"
    click BenchEntry "https://github.com/nanonets/docext/blob/main/docext/benchmark/benchmark.py"
    click TaskRunner "https://github.com/nanonets/docext/blob/main/docext/benchmark/tasks.py"
    click Dataset "https://github.com/nanonets/docext/tree/main/docext/benchmark/vlm_datasets/"
    click MetricsClassification "https://github.com/nanonets/docext/blob/main/docext/benchmark/metrics/classification.py"
    click MetricsKIE "https://github.com/nanonets/docext/blob/main/docext/benchmark/metrics/kie.py"
    click MetricsOCR "https://github.com/nanonets/docext/blob/main/docext/benchmark/metrics/ocr.py"
    click MetricsTables "https://github.com/nanonets/docext/blob/main/docext/benchmark/metrics/tables.py"
    click MetricsVQA "https://github.com/nanonets/docext/blob/main/docext/benchmark/metrics/vqa.py"
    click UtilsBench "https://github.com/nanonets/docext/blob/main/docext/benchmark/utils.py"
    click Docker "https://github.com/nanonets/docext/tree/main/Dockerfile"
    click ConfigFiles "https://github.com/nanonets/docext/blob/main/configs/benchmark.yaml"
    click Setup "https://github.com/nanonets/docext/blob/main/setup.py"

    %% Styles
    classDef frontend fill:#AED6F1,stroke:#1F618D,color:#1B4F72
    classDef core fill:#A9DFBF,stroke:#196F3D,color:#145A32
    classDef bench fill:#F9E79F,stroke:#B7950B,color:#7D6608
    classDef external fill:#F5B7B1,stroke:#C0392B,color:#78281F
    classDef config fill:#D7BDE2,stroke:#6C3483,color:#4A235A
```
