# enoch3712/extractthinker architecture

> Architecture diagram of the GitHub repository enoch3712/extractthinker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/enoch3712/extractthinker
- Repository: https://github.com/enoch3712/extractthinker

## Overview

##### Main Components

a) Core Processing Components:
- Document Loaders Hub
- Extractor Engine
- Classification System
- Splitter System
- Contract Management
- LLM Integration Layer

b) External Integrations:
- Multiple LLM Providers (OpenAI, Anthropic, Cohere, Azure OpenAI, Local/Ollama)
- Document Processing Services (AWS Textract, Azure Form Recognizer, Google Document AI)
- OCR Services (Tesseract)

##### Input Layer
- Document input handlers (PDF, Images, Spreadsheets, Text)
- File system interfaces
- Stream handlers

##### Document Processing Layer
- Document Loader Hub (showing all supported loaders)
- Caching system
- Pre-processing components

##### Core Processing Layer
- Classification Engine
- Extraction Engine
- Splitting System
- Contract Management

##### LLM Integration Layer
- LLM Router
- Provider-specific adapters
- Response handlers

##### Output Layer
- Contract-based structured data
- Batch processing results
- Classification results

##### Key Relationships to Highlight

- Synchronous vs. Asynchronous flows
- Data transformation paths
- Classification decision trees
- Contract validation flows
- Batch processing workflows

##### Specific Implementation Details

- The modular architecture allowing multiple document loaders
- The contract-based extraction system using Pydantic models
- The classification tree structure
- Splitting strategies (Lazy vs. Eager)
- Batch processing capabilities
- LLM provider abstraction

##### Component Details

Document Loader Hub:
**┌─────────────────────┐**
- Document Loader Hub│
- ┤
- PyPDF            │
- Tesseract        │
- AWS Textract     │
- Azure Form       │
- Google Doc AI    │
- ┘

Extraction Pipeline:
**┌─────────────┐    ┌──────────────┐    ┌──────────────┐**
- Document   │ -> │ Classification│ -> │  Extraction  │
- Input     │    │    Engine    │    │    Engine    │
- ┘    └──────────────┘    └──────────────┘

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef core fill:#2196F3,stroke:#1565C0,color:white
    classDef external fill:#4CAF50,stroke:#2E7D32,color:white
    classDef contract fill:#FFC107,stroke:#FFA000,color:black
    classDef critical fill:#F44336,stroke:#D32F2F,color:white
    classDef llm fill:#9C27B0,stroke:#6A1B9A,color:white
    
    %% Input Layer
    subgraph Input
        DI[Document Input]
        FS[File System Interface]
        SH[Stream Handler]
    end

    %% Document Processing Layer
    subgraph DocProcessing
        DLH[Document Loaders Hub]:::core
        Cache[Caching System]
        PreProc[Pre-processing]
        
        subgraph Loaders
            Tesseract[Tesseract OCR]:::external
            AWSTextract[AWS Textract]:::external
            AzureForm[Azure Form Recognizer]:::external
            GoogleDoc[Google Document AI]:::external
        end
    end

    %% Core Processing Layer
    subgraph CoreProcessing
        CE[Classification Engine]:::core
        EE[Extraction Engine]:::core
        SS[Splitting System]:::core
        CM[Contract Management]:::contract
        BP[Batch Processing]:::core
    end

    %% LLM Integration Layer
    subgraph LLMLayer
        LLMR[LLM Router]:::llm
        CH[Completion Handler]:::llm
        
        subgraph LLMProviders
            OpenAI[OpenAI]:::external
            Anthropic[Anthropic]:::external
            Cohere[Cohere]:::external
            AzureOpenAI[Azure OpenAI]:::external
            Ollama[Local/Ollama]:::external
        end
    end

    %% Output Layer
    subgraph Output
        SD[Structured Data]
        BR[Batch Results]
        CR[Classification Results]
    end

    %% Relationships
    DI --> DLH
    FS --> DLH
    SH --> DLH
    
    DLH --> Tesseract
    DLH --> AWSTextract
    DLH --> AzureForm
    DLH --> GoogleDoc
    
    DLH --> PreProc
    PreProc --> Cache
    Cache --> CoreProcessing
    
    CE --> EE
    EE --> SS
    SS --> CM
    CM --> BP
    
    CoreProcessing --> LLMR
    LLMR --> CH
    CH --> LLMProviders
    
    CoreProcessing --> Output

    %% Click Events
    click DLH "https://github.com/enoch3712/ExtractThinker/tree/main/extract_thinker/document_loader/"
    click EE "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/extractor.py"
    click CE "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/models/classification.py"
    click SS "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/splitter.py"
    click CM "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/models/contract.py"
    click LLMR "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/llm.py"
    click CH "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/completion_handler.py"
    click AWSTextract "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/document_loader/document_loader_aws_textract.py"
    click AzureForm "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/document_loader/document_loader_azure_document_intelligence.py"
    click GoogleDoc "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/document_loader/document_loader_google_document_ai.py"
    click Tesseract "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/document_loader/document_loader_tesseract.py"
    click BP "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/batch_job.py"
    click CoreProcessing "https://github.com/enoch3712/ExtractThinker/blob/main/extract_thinker/process.py"

    %% Legend
    subgraph Legend
        L1[Core Components]:::core
        L2[External Services]:::external
        L3[Contract Components]:::contract
        L4[Critical Path]:::critical
        L5[LLM Components]:::llm
    end
```
