How privateai-com/docviz works
Project Type and Purpose
• This is an open-source Python library/tool (not a web app) whose sole purpose is to extract structured content from documents (primarily PDFs) via a simple API or CLI.
• Key features include streaming extraction, batch processing, selective extraction, pluggable detection backends (YOLO-based), optional LLM-based summarization, and multi-format export (JSON, CSV, Excel, etc.).
High-Level Architecture and Main Components
A. User Interface Layer
• CLI entrypoint (src/docviz/cli/__main__.py)
• Python API entrypoint (docviz.Document and associated functions)
B. Core Library Layer
• Document class (src/docviz/lib/document) – the façade that accepts a file/URL and exposes extract_content()/extract_content_sync() and streaming methods.
• Batch extract utilities (docviz.batch_extract)
C. Pipeline Orchestration Layer
• ExtractionPipeline (src/docviz/lib/extraction/pipeline.py) – coordinates the end-to-end flow for each page or document.
• Configuration subsystem (ExtractionConfig, LLMConfig in src/docviz/types)
• Environment/config cache (src/docviz/config_cache.py, environment.py)
D. Subsystems
1. PDF Processing
– Converter & analyzer (src/docviz/lib/pdf/convert.py, pdf_analyzer.py, text_extraction.py)
2. Detection
– Detection front-end (frontend.py) picks a backend based on config
– Backends (doclayout_yolo.py, yolo_doclaynet.py)
– Deduplication (deduplication.py)
3. Content Extraction
– Image preprocessing & annotation (src/docviz/lib/image)
– Summarizer (image/summarizer.py)
– Document utilities (class_.py, utils.py)
4. Caching
– Model cache (src/docviz/model_cache.py) for YOLO weights (models/*.pt)
– Config cache for repeated runs
E. Types & Output Layer
• Typed models for inputs and outputs (src/docviz/types/*.py)
• SaveFormat and save routines for JSON, CSV, Excel, etc.
F. External Services & Artifacts
• Pre-trained model files (models/*.pt)
• Optional LLM API calls (e.g., OpenAI via llm_config)
Relationships and Data Flow
1. User invokes CLI or Python API →
2. Document object created →
3. Document calls ExtractionPipeline
4. Pipeline uses PDF Processor → Detection subsystem → Deduplication → Extraction modules → (optionally) LLM summarizer
5. Intermediate results conform to typed schemas →
6. Final results passed to save routines → filesystem or other sinks
7. Throughout: ModelCache and ConfigCache provide weights and settings; environment.py manages global settings.
Architectural Patterns and Design Principles
• Layered architecture separating interface, orchestration, subsystems, and output.
• Plugin-style detection backends for easy extension.
• Typed data models for strong contract between subsystems.
• Caching abstraction to minimize expensive startup costs.
• Single Responsibility: each package (pdf, detection, image, extraction) encapsulates its own logic.