How yuezhengmeng/graphrag-learn works
Main Components
• Input & Data Ingestion: Represent the data source layer where raw text files (e.g., data/kongyiji.txt and inputs in the “kongyiji” directory) are loaded. Include “kongyiji/input” and “kongyiji/prompts” as configuration and prompt resources.
• Config & Settings Management: Show the configuration layer that handles diverse input models, configuration files (located in graphrag/config and extra_data/settings.yaml) to control parameters for the LLM, caching, chunking, etc.
• Preprocessing and text splitting (graphrag/index/text_splitting)
• Graph construction and update (graphrag/index/graph, which covers embedding, extraction, clustering, layout, merge, and visualization modules)
• Workflows that drive the creation of various artifacts such as base documents, entities, relationships, and community reports (under graphrag/index/workflows/v1).
• Query Engine: Represent the querying subsystem (from graphrag/query). This separates into:
• Local Search (using internal vector stores and context builders)
• Global Search (likely using a combination of system prompts and external retrieval techniques)
• Vector Stores: Include a component for vector storage (graphrag/vector_stores) where embeddings are managed—this supports efficient similarity searches during queries.
• Visualization Interface: Illustrate the “visualize_graph” folder as a separate component. This uses notebooks and HTML artifacts (with caches of CSS and JS assets) to render interactive graphs via pyvis.
• Supporting Utilities & Caches: Represent supporting libraries for caching (graphrag/index/cache and caching mechanisms in the LLM layer), progress tracking, error handling, and other common utilities across the system.
Relationships and Data Flow
• Data flows from the Input Layer into preprocessing modules (text splitting and extraction). From there, the system applies LLM interactions for entity extraction and summarization.
• The processed data is then passed to the graph building pipeline where nodes (entities, text units, documents) and edges (relationships) are created.
• Once the graph is built and stored as artifacts (parquet files under kongyiji/output/*), the Query Engine (with its two methods: local and global) can access this data. Queries trigger context builders to interact with LLM module for retrieval.
• The Visualization Interface receives graph data from the stored artifacts and renders it interactively for user exploration.
Architectural Patterns & Design Principles
• Modular Pipelines: The codebase is organized as a set of decoupled modules (each under its own directory, e.g., extraction, embedding, workflows), following the principle of separation of concerns.
• Configuration-Driven Execution: The entire system’s behavior is highly determined by configuration files and YAML settings, making it extensible and customizable.
• LLM Abstraction: The LLM calls are wrapped in rate-limiting, history tracking, and caching layers to ensure robustness.
• Extensibility & Workflow Orchestration: Using workflow definitions (graphrag/index/workflows) allows clear control over different processing stages that can be independently updated or extended.
Relevant Technologies & Frameworks
• Python environment with dependencies listed in requirements.txt (with a note on uvloop exclusion on Windows).
• LLM integrations through OpenAI API clients and potentially Chinese LLM support (Qwen, Tongyi).
• Visualization using “pyvis” (with caching of external assets in visualize_graph/lib).