# privateai-com/docviz architecture

> Architecture diagram of the GitHub repository privateai-com/docviz, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/privateai-com/docviz
- Repository: https://github.com/privateai-com/docviz

## Overview

##### Project Type and Purpose
- This is an open-source Python library/tool (not a web app) whose sole purpose is to extract structured content from documents (primarily PDFs) via a simple API or CLI.
- Key features include streaming extraction, batch processing, selective extraction, pluggable detection backends (YOLO-based), optional LLM-based summarization, and multi-format export (JSON, CSV, Excel, etc.).

##### High-Level Architecture and Main Components

A. User Interface Layer
- CLI entrypoint (src/docviz/cli/__main__.py)
- Python API entrypoint (docviz.Document and associated functions)

B. Core Library Layer
- Document class (src/docviz/lib/document) – the façade that accepts a file/URL and exposes extract_content()/extract_content_sync() and streaming methods.
- Batch extract utilities (docviz.batch_extract)

C. Pipeline Orchestration Layer
- ExtractionPipeline (src/docviz/lib/extraction/pipeline.py) – coordinates the end-to-end flow for each page or document.
- Configuration subsystem (ExtractionConfig, LLMConfig in src/docviz/types)
- Environment/config cache (src/docviz/config_cache.py, environment.py)

D. Subsystems
1. PDF Processing
– Converter & analyzer (src/docviz/lib/pdf/convert.py, pdf_analyzer.py, text_extraction.py)
2. Detection
– Detection front-end (frontend.py) picks a backend based on config
– Backends (doclayout_yolo.py, yolo_doclaynet.py)
– Deduplication (deduplication.py)
3. Content Extraction
– Image preprocessing & annotation (src/docviz/lib/image)
– Summarizer (image/summarizer.py)
– Document utilities (class_.py, utils.py)
4. Caching
– Model cache (src/docviz/model_cache.py) for YOLO weights (models/*.pt)
– Config cache for repeated runs

E. Types & Output Layer
- Typed models for inputs and outputs (src/docviz/types/*.py)
- SaveFormat and save routines for JSON, CSV, Excel, etc.

F. External Services & Artifacts
- Pre-trained model files (models/*.pt)
- Optional LLM API calls (e.g., OpenAI via llm_config)

##### Relationships and Data Flow
1. User invokes CLI or Python API →
2. Document object created →
3. Document calls ExtractionPipeline
4. Pipeline uses PDF Processor → Detection subsystem → Deduplication → Extraction modules → (optionally) LLM summarizer
5. Intermediate results conform to typed schemas →
6. Final results passed to save routines → filesystem or other sinks
7. Throughout: ModelCache and ConfigCache provide weights and settings; environment.py manages global settings.

##### Architectural Patterns and Design Principles
- Layered architecture separating interface, orchestration, subsystems, and output.
- Plugin-style detection backends for easy extension.
- Typed data models for strong contract between subsystems.
- Caching abstraction to minimize expensive startup costs.
- Single Responsibility: each package (pdf, detection, image, extraction) encapsulates its own logic.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% UI Layer
    subgraph "User Interface" 
        direction TB
        CLI["CLI Entrypoint"]:::ui
        API["Document API Entrypoint"]:::ui
    end

    %% Core Library Layer
    subgraph "Core Library" 
        direction TB
        Document["Document Facade"]:::core
        Batch["Batch Extraction Utilities"]:::core
    end

    %% Orchestration Layer
    subgraph "Pipeline Orchestration" 
        direction TB
        Pipeline["ExtractionPipeline"]:::orchestration
        subgraph "Configuration" 
            direction TB
            ExtConfig["ExtractionConfig"]:::orchestration
            LLMConfig["LLMConfig"]:::orchestration
        end
        EnvCache["Env & Config Cache"]:::cache
        ModelCache["Model Cache"]:::cache
    end

    %% Subsystems Layer
    subgraph "Processing Subsystems" 
        direction TB
        subgraph "PDF Processor" 
            direction TB
            Converter["Converter"]:::subsystem
            Analyzer["PDF Analyzer"]:::subsystem
            TextExtract["Text Extraction"]:::subsystem
        end
        subgraph "Detection Subsystem" 
            direction TB
            Frontend["Detection Frontend"]:::subsystem
            Dedup["Deduplication"]:::subsystem
            subgraph "Backends" 
                direction TB
                BaseBE["Base Backend"]:::subsystem
                YOLO1["doclayout_yolo"]:::subsystem
                YOLO2["yolo_doclaynet"]:::subsystem
            end
        end
        subgraph "Content Extraction" 
            direction TB
            Preproc["Image Preprocessing"]:::subsystem
            Annotate["Annotation"]:::subsystem
            Summarizer["Summarizer"]:::subsystem
        end
    end

    %% Types & Output Layer
    subgraph "Types & Output" 
        direction TB
        TypedModels["Typed Models"]:::types
        SaveUtils["Save & Format Utilities"]:::types
    end

    %% External Services & Artifacts
    subgraph "External" 
        direction TB
        ModelsDB["Pre-trained Models"]:::external
        LLMAPI["LLM API"]:::external
    end

    %% Connections
    CLI -->|invokes| Document
    API -->|invokes| Document
    Document -->|uses| Pipeline
    Document -->|uses| Batch
    Pipeline -->|loads| ExtConfig
    Pipeline -->|loads| LLMConfig
    Pipeline -->|uses| EnvCache
    Pipeline -->|uses| ModelCache
    Pipeline -->|orchestrates| Converter
    Pipeline -->|orchestrates| Analyzer
    Pipeline -->|orchestrates| TextExtract
    Pipeline -->|orchestrates| Frontend
    Frontend -->|calls| BaseBE
    Frontend -->|calls| YOLO1
    Frontend -->|calls| YOLO2
    Frontend -->|passes to| Dedup
    Dedup -->|passes to| Preproc
    Preproc -->|annotates| Annotate
    Annotate -->|summarizes| Summarizer
    Summarizer -->|returns| Pipeline
    Pipeline -->|emits| TypedModels
    TypedModels -->|feeds| SaveUtils
    SaveUtils -->|writes to| ModelsDB
    Pipeline -->|optional LLM call| LLMAPI

    %% Click Events
    click CLI "https://github.com/privateai-com/docviz/blob/main/src/docviz/cli/__main__.py"
    click API "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/document/class_.py"
    click Document "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/document/class_.py"
    click Batch "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/functions.py"
    click Pipeline "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/extraction/pipeline.py"
    click SaveUtils "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/extraction/utils.py"
    click ExtConfig "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/extraction_config.py"
    click LLMConfig "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/llm_config.py"
    click EnvCache "https://github.com/privateai-com/docviz/blob/main/src/docviz/environment.py"
    click EnvCache "https://github.com/privateai-com/docviz/blob/main/src/docviz/config_cache.py"
    click ModelCache "https://github.com/privateai-com/docviz/blob/main/src/docviz/model_cache.py"
    click Converter "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/pdf/convert.py"
    click Analyzer "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/pdf/pdf_analyzer.py"
    click TextExtract "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/pdf/text_extraction.py"
    click Frontend "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/detection/frontend.py"
    click Dedup "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/detection/deduplication.py"
    click BaseBE "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/detection/backends/base.py"
    click YOLO1 "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/detection/backends/doclayout_yolo.py"
    click YOLO2 "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/detection/backends/yolo_doclaynet.py"
    click Preproc "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/image/preprocessing.py"
    click Annotate "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/image/annotate.py"
    click Summarizer "https://github.com/privateai-com/docviz/blob/main/src/docviz/lib/image/summarizer.py"
    click TypedModels "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/aliases.py"
    click TypedModels "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/detection_config.py"
    click TypedModels "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/detection_result.py"
    click TypedModels "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/extraction_chunk.py"
    click TypedModels "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/extraction_result.py"
    click TypedModels "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/extraction_type.py"
    click TypedModels "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/includes_preset.py"
    click TypedModels "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/labels.py"
    click TypedModels "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/ocr_config.py"
    click SaveUtils "https://github.com/privateai-com/docviz/blob/main/src/docviz/types/save_format.py"

    %% Styles
    classDef ui fill:#D0E8FF,stroke:#0366D6,color:#0366D6
    classDef core fill:#B3E5FC,stroke:#0288D1,color:#01579B
    classDef orchestration fill:#B2DFDB,stroke:#00796B,color:#004D40
    classDef subsystem fill:#C8E6C9,stroke:#388E3C,color:#1B5E20
    classDef cache fill:#EEEEEE,stroke:#9E9E9E,color:#424242
    classDef types fill:#E1BEE7,stroke:#8E24AA,color:#4A148C
    classDef external fill:#FFF9C4,stroke:#FBC02D,color:#F57F17
```
