# documindhq/documind architecture

> Architecture diagram of the GitHub repository documindhq/documind, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/documindhq/documind
- Repository: https://github.com/documindhq/documind

## Overview

##### Project type and purpose
- Documind is a modular Node.js document-processing library/CLI service.
- It ingests PDFs (and other formats), converts them to text, optionally auto-generates schemas, sends prompts to LLM providers, and formats structured JSON/Markdown output.

##### File structure
- Top-level:
– `core/`: abstraction over LLM providers (OpenAI, Google, Ollama), common types and utils.
– `extractor/`: document-pipeline implementation (conversion, schema generation, extraction, formatting, templates).
- Key patterns: provider-strategy pattern in `core/src/providers`; a staged pipeline in `extractor/src`: converters → extractors → services → templates → formatters.
- Config/build: TS in core, plain JS in extractor; `.env` for API keys; Ghostscript/GraphicsMagick external deps.

##### README for extra insights
- Core features: PDF extraction, custom schemas, auto-schema, object/array support, Markdown formatting, plugin-style templates.
- Supports cloud (OpenAI/Google) and local LLMs (Llava, Llama3.2-vision).

1. Entry Point (CLI or `extract()` API)
2. Preprocessor
- File Validator
- Converter (Ghostscript + GraphicsMagick) → raw text
3. Schema Manager
- User-defined schema loader
- Auto-schema generator (optional)
4. Extraction Engine
- Prompt builder (prompts.js)
- Core provider adapter (core)
5. Postprocessor
- Formatter (JSON, Markdown)
- Template manager (built-in templates)
6. Output (structured JSON/MD)

- File in → Preprocessor → Text out → (Schema Manager) → Extraction Engine → Core Providers → LLM APIs/Local LLM → Extraction Engine receives response → Postprocessor → Final Output.

c. Architectural patterns & principles:
- Strategy pattern for provider plugins (core/src/providers)
- Pipeline stages (validate → convert → extract → format)
- Separation of concerns: core handles LLM logic, extractor handles document logic.
- Extensibility via templates and provider adapters.

d. Key technologies & libraries:
- Node.js + TypeScript (core), JavaScript (extractor)
- Ghostscript & GraphicsMagick (native deps)
- OpenAI, Google Cloud LLM, Ollama, local LLM servers
- Zod schemas for validation (in utils)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Extractor Module
    subgraph "Extractor" 
        direction TB
        EntryPoint["Entry Point (CLI/API)"]:::extractorModule
        FileValidator["File Validator"]:::extractorModule
        PDFValidator["PDF Validator"]:::extractorModule
        Converter["Converter\n(Ghostscript/GraphicsMagick)"]:::extractorModule
        subgraph "Schema Manager"
            direction TB
            UserSchema["User-defined Schema Loader"]:::extractorModule
            AutoSchemaGen["Auto-Schema Generator"]:::extractorModule
            CleanSchema["Schema Cleaner"]:::extractorModule
            GenSchemas["Generation Schemas"]:::extractorModule
            ZodSchema["Convert to Zod Schema"]:::extractorModule
            SchemaValidator["Schema Validator"]:::extractorModule
        end
        subgraph "Extraction Engine"
            direction TB
            PromptBuilder["Prompt Builder"]:::extractorModule
            ExtractorsIndex["Provider-Specific Extractors\n(Index)"]:::extractorModule
            OpenAIExtractor["OpenAI Extractor"]:::extractorModule
            GoogleExtractor["Google Extractor"]:::extractorModule
            OllamaExtractor["Ollama Extractor"]:::extractorModule
        end
        subgraph "Postprocessor"
            direction TB
            Formatter["Formatting Service"]:::extractorModule
            MarkdownUtil["Generate Markdown"]:::extractorModule
            TemplatesService["Template Manager"]:::extractorModule
            TemplatesDir["Built-in Templates"]:::extractorModule
        end
        Output["Output\n(Structured JSON/Markdown)"]:::extractorModule
    end

    %% Core Module
    subgraph "Core" 
        direction TB
        CoreEntry["Core Library Entry"]:::coreModule
        subgraph "Providers"
            OpenAIAdapter["OpenAI Adapter"]:::coreModule
            GoogleAdapter["Google Adapter"]:::coreModule
            OllamaAdapter["Ollama Adapter"]:::coreModule
            CompletionUtil["Completion Util"]:::coreModule
        end
    end

    %% External Services
    Ghostscript["Ghostscript/GraphicsMagick"]:::external
    LLMEndpoints["LLM Endpoints\n(OpenAI, Google, Ollama, Local)"]:::external

    %% Flow
    EntryPoint -->|"file"| FileValidator
    FileValidator --> PDFValidator
    PDFValidator -->|"validated file"| Converter
    Converter -->|"text"| UserSchema
    Converter -->|"text"| AutoSchemaGen
    AutoSchemaGen --> CleanSchema
    CleanSchema --> GenSchemas
    GenSchemas --> ZodSchema
    ZodSchema --> SchemaValidator
    SchemaValidator --> PromptBuilder
    UserSchema --> PromptBuilder
    PromptBuilder --> ExtractorsIndex
    ExtractorsIndex --> OpenAIExtractor
    ExtractorsIndex --> GoogleExtractor
    ExtractorsIndex --> OllamaExtractor
    OpenAIExtractor -->|"prompt"| OpenAIAdapter
    GoogleExtractor -->|"prompt"| GoogleAdapter
    OllamaExtractor -->|"prompt"| OllamaAdapter
    OpenAIAdapter --> CompletionUtil
    GoogleAdapter --> CompletionUtil
    OllamaAdapter --> CompletionUtil
    CompletionUtil --> LLMEndpoints
    LLMEndpoints --> CompletionUtil
    CompletionUtil --> PromptBuilder
    PromptBuilder --> Formatter
    Formatter --> MarkdownUtil
    Formatter --> TemplatesService
    TemplatesService --> TemplatesDir
    MarkdownUtil --> Output
    TemplatesService --> Output

    %% Click Events
    click EntryPoint "https://github.com/documindhq/documind/blob/main/extractor/src/index.js"
    click CoreEntry "https://github.com/documindhq/documind/blob/main/core/src/index.ts"
    click FileValidator "https://github.com/documindhq/documind/blob/main/extractor/src/utils/fileValidator.js"
    click PDFValidator "https://github.com/documindhq/documind/blob/main/extractor/src/utils/pdfValidator.js"
    click Converter "https://github.com/documindhq/documind/blob/main/extractor/src/utils/convertToText.js"
    click AutoSchemaGen "https://github.com/documindhq/documind/blob/main/extractor/src/autoschema/autogenerateSchema.js"
    click CleanSchema "https://github.com/documindhq/documind/blob/main/extractor/src/autoschema/cleanSchemaFields.js"
    click GenSchemas "https://github.com/documindhq/documind/tree/main/extractor/src/autoschema/generation-schemas/"
    click ZodSchema "https://github.com/documindhq/documind/blob/main/extractor/src/utils/convertToZodSchema.js"
    click SchemaValidator "https://github.com/documindhq/documind/blob/main/extractor/src/utils/schemaValidator.js"
    click PromptBuilder "https://github.com/documindhq/documind/blob/main/extractor/src/prompts.js"
    click OpenAIAdapter "https://github.com/documindhq/documind/blob/main/core/src/providers/openAI.ts"
    click GoogleAdapter "https://github.com/documindhq/documind/blob/main/core/src/providers/google.ts"
    click OllamaAdapter "https://github.com/documindhq/documind/blob/main/core/src/providers/ollama.ts"
    click CompletionUtil "https://github.com/documindhq/documind/blob/main/core/src/providers/utils/completion.ts"
    click OpenAIExtractor "https://github.com/documindhq/documind/blob/main/extractor/src/extractors/openAI.js"
    click GoogleExtractor "https://github.com/documindhq/documind/blob/main/extractor/src/extractors/google.js"
    click OllamaExtractor "https://github.com/documindhq/documind/blob/main/extractor/src/extractors/ollama.js"
    click ExtractorsIndex "https://github.com/documindhq/documind/blob/main/extractor/src/extractors/index.js"
    click Formatter "https://github.com/documindhq/documind/blob/main/extractor/src/services/formatter.js"
    click MarkdownUtil "https://github.com/documindhq/documind/blob/main/extractor/src/utils/generateMarkdown.js"
    click TemplatesService "https://github.com/documindhq/documind/blob/main/extractor/src/services/templates.js"
    click TemplatesDir "https://github.com/documindhq/documind/tree/main/extractor/src/templates/"

    %% Styles
    classDef coreModule fill:#BBDEFB,stroke:#444,stroke-width:1px
    classDef extractorModule fill:#C8E6C9,stroke:#444,stroke-width:1px
    classDef external fill:#FFE0B2,stroke:#444,stroke-width:1px
```
