How stair-lab/kg-gen works
This project is an open-source Python library—kg-gen—that extracts knowledge graphs from plain text using AI language models. The primary use case is to transform unstructured textual input (which can be a plain string or an array of messages) into a structured graph representing entities and their relationships. The system can also perform additional steps like chunking long texts, aggregating multiple graph outputs, and clustering similar entities/relations.
• KGGen Core Module: Emphasize the central class (KGGen in src/kg_gen/kg_gen.py) which orchestrates graph generation.
• Processing Stages (Steps): Represent the three core processing scripts:
• _1_get_entities.py – responsible for entity extraction
• _2_get_relations.py – responsible for identifying relationships between the entities
• _3_cluster_graph.py – handles clustering to merge similar entities or relations
• Utility Functions: Outline the auxiliary utilities (e.g., chunk_text in src/kg_gen/utils) that support tasks like text chunking.
• External Model Integration: Show a component that represents connections to external AI/LLM services via LiteLLM. Note that calls are made to model providers such as OpenAI, Ollama, Anthropic, and others.
• Graph Aggregation and Clustering Services: Include nodes that process multiple graphs (aggregate method) and further refine clusters (cluster method).
• Testing and Benchmarking: Though not a runtime component, include a section for tests (tests directory) and the benchmark use-case in the MINE folder to indicate quality assurance and experimental validation.
Relationships and Interactions
• Incoming text data flows from the Input Interface into the KGGen Core Module.
• Within KGGen, the system sequentially calls the processing steps (get entities, get relations, and then clustering) in the order listed in the “steps” folder.
• The component for External Model Integration should be connected both to the KGGen Core and to the processing steps, indicating that model calls (routed via LiteLLM) support the extraction and clustering functions.
Architectural Patterns and Principles
• Modular Design: Each functionality (extraction, relation detection, clustering, utility functions) is split into separate modules, promoting separation of concerns.
• Pipeline/Sequential Processing: The project processing is best represented as a pipeline with clearly defined sequential steps.
• Extensibility: Design elements (e.g., support for multiple external model providers) make the system highly adaptable.
Technological Details and External Libraries
• That the system leverages AI models through API calls handled by LiteLLM.
• DSPy as the technology used for structured output generation.
• Mention Python packaging components (pyproject.toml, dist/) which signal that the system is packaged as a pip-installable module.