How documindhq/documind works
Project type and purpose
• Documind is a modular Node.js document-processing library/CLI service.
• It ingests PDFs (and other formats), converts them to text, optionally auto-generates schemas, sends prompts to LLM providers, and formats structured JSON/Markdown output.
File structure
• Top-level:
– core/: abstraction over LLM providers (OpenAI, Google, Ollama), common types and utils.
– extractor/: document-pipeline implementation (conversion, schema generation, extraction, formatting, templates).
• Key patterns: provider-strategy pattern in core/src/providers; a staged pipeline in extractor/src: converters → extractors → services → templates → formatters.
• Config/build: TS in core, plain JS in extractor; .env for API keys; Ghostscript/GraphicsMagick external deps.
README for extra insights
• Core features: PDF extraction, custom schemas, auto-schema, object/array support, Markdown formatting, plugin-style templates.
• Supports cloud (OpenAI/Google) and local LLMs (Llava, Llama3.2-vision).
1. Entry Point (CLI or extract() API)
2. Preprocessor
• File Validator
• Converter (Ghostscript + GraphicsMagick) → raw text
3. Schema Manager
• User-defined schema loader
• Auto-schema generator (optional)
4. Extraction Engine
• Prompt builder (prompts.js)
• Core provider adapter (core)
5. Postprocessor
• Formatter (JSON, Markdown)
• Template manager (built-in templates)
6. Output (structured JSON/MD)
• File in → Preprocessor → Text out → (Schema Manager) → Extraction Engine → Core Providers → LLM APIs/Local LLM → Extraction Engine receives response → Postprocessor → Final Output.
c. Architectural patterns & principles:
• Strategy pattern for provider plugins (core/src/providers)
• Pipeline stages (validate → convert → extract → format)
• Separation of concerns: core handles LLM logic, extractor handles document logic.
• Extensibility via templates and provider adapters.
d. Key technologies & libraries:
• Node.js + TypeScript (core), JavaScript (extractor)
• Ghostscript & GraphicsMagick (native deps)
• OpenAI, Google Cloud LLM, Ollama, local LLM servers
• Zod schemas for validation (in utils)