How auvalab/itext2kg works
Main Components
• Document Distiller: This module takes raw documents and processes them into semantic blocks based on a user-defined schema. It is responsible for parsing information from scientific papers, CVs, company websites, and other inputs.
• Incremental Entity Extractor: It extracts and disambiguates entities from the refined document sections. It uses cosine similarity-based matching and applies configurable weights.
• Incremental Relation Extractor: This component identifies relationships between these entities, working on both local and global context. It also handles LLM-specific issues such as hallucination or missing relations.
• Graph Integrator & Visualization: Responsible for integrating the extracted components into a Neo4j database, providing a graph-based visualization for exploration and interactive analysis.
• Utility and Supporting Modules: Includes utility functions such as the LLM output parser, matcher, and schema definitions that support the main modules.
• External Integrations: Notice that the system utilizes external LLMs and embedding models (for instance, OpenAI, Mistral, and LangChain integrations) which are critical for the extraction processes.
Representing Relationships and Interactions
• Raw Documents (inputs) → Document Distiller.
• Distilled semantic blocks → Incremental Entity Extractor.
• Distilled semantic blocks (or extracted entities) → Incremental Relation Extractor.
• Both the extracted entities and relations feed into the core iText2KG engine, which orchestrates these operations.
• The results, in the form of a knowledge graph, are then passed to the Graph Integrator for visualization and further interaction via Neo4j.
• External dependencies:
• Data flow from the Knowledge Graph (or constructed semantic representations) into the Neo4j database.
Architectural Patterns and Design Principles
• Modular Separation – Each key functionality is encapsulated in its submodule (distiller, extraction modules, integrator). Indicate these modules as distinct blocks.
• Incremental Processing – The graph is built in an incremental manner where new documents can trigger updates to the existing knowledge graph.
• External Services Integration – The design is extensible by integrating multiple external model services. Represent these as “External LLM Service”, “External Embedding Service”, and “Neo4j Graph Database”.
• Input Transformation – Begin with raw text and apply a transformation pipeline (distillation → extraction → graph integration).
Including Relevant Technologies and Frameworks
• Python Environment with dependencies indicated in pyproject.toml, requirements.txt, and setup.cfg.
• Neo4j for graph database storage and visualization.
• LangChain integration for LLM and embedding models.
• pydantic (as seen from schema definitions) for data modeling.