How rahulnyk/knowledge_graph_maker works
Main components
• User Input / Document Ingestion: The starting point where user texts are provided. This may include functionality for splitting texts into chunks and wrapping them into Document models (using pydantic).
• Ontology Definition Module: Shows that the ontology is defined as a pydantic model with prescribed labels and relationships.
• GraphMaker Engine: The core module (found in graph_maker.py) that takes documents, processes each chunk, calls the LLM for graph generation, and parses responses into graph edges (with Node and Edge pydantic models).
• LLM Client Abstraction: An abstraction representing the LLM client interface. This module supports multiple clients (OpenAIClient, GroqClient, OllamaClient) and follows a common abstract base class design.
• Logging Component: The logger component used across the library for tone and debugging information.
• Neo4j Integration Module (Optional): Represented by Neo4jGraphModel, which takes the generated graph edges and saves them into a Neo4j database (supporting index creation, transactional saving, etc.).
• Distribution/Packaging Component: Not directly part of runtime architecture but important for deployment and distribution (indicated by the “dist” folder and packaging files).
Relationships and Data Flows
• The text is split into chunks; each chunk is converted into a Document model that also carries metadata.
• Documents are fed to the GraphMaker engine that:
• Uses the defined Ontology as a prompt guide.
• Iterates through the Documents.
• For each one, it sends the text (with predefined prompt tuning) to the LLM client.
• The LLM client generates a response, which the GraphMaker then parses. Parsing includes handling JSON issues and splitting responses appropriately.
• Successfully parsed responses yield a list of Edge models (with embedded Node objects and metadata).
• Optionally, if configured, the graph is passed on to the Neo4j Integration Module, which handles indexing and saving into a Neo4j graph database.
• Logging occurs concurrently to document key steps (using the logger module).
Architectural Patterns and Design Principles
• Modularity: Each core functionality (document processing, LLM integration, graph creation, Neo4j persistence) is encapsulated in its own module.
• Abstraction: The LLM Client uses an abstract base class allowing extensibility (users can add their own LLM clients as long as they implement the abstract methods).
• Robust Error Handling and Resilience: The GraphMaker is designed to handle faulty LLM responses (e.g., malformed JSON) by applying auto-correction and multiple parsing attempts.
• Extendability: The use of pydantic models for ontology, documents, and graph components makes it easy to extend or modify the schemas without significant refactoring.
Relevant Technologies, Frameworks, or Libraries
• Python, with specific usage of pydantic for data modeling.
• LLM APIs (OpenAI, Groq) integrated as client libraries.
• Neo4j for graph database persistence.
• Packaging tools (pyproject.toml, poetry) for dependency management and distribution.
• GitHub Actions for CI/CD (as seen from the .github/workflows directory).
Instructing the Principal Software Engineer
• A Parser component that transforms LLM output into structured graph edges/models.
• An optional branch where the graph data is sent to the “Neo4j Integration Module,” indicating a clear dependency for storing and optionally visualizing the constructed graph.
• A small auxiliary module for “Logging & Error Management” that interacts with the GraphMaker Engine.
• Data flow from documents to GraphMaker.
• Communication from GraphMaker to the LLM Clients.
• The resulting flow from GraphMaker to the Neo4j module.