# yuezhengmeng/graphrag-learn architecture

> Architecture diagram of the GitHub repository yuezhengmeng/graphrag-learn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yuezhengmeng/graphrag-learn
- Repository: https://github.com/yuezhengmeng/graphrag-learn

## Overview

##### Main Components
- Input & Data Ingestion: Represent the data source layer where raw text files (e.g., data/kongyiji.txt and inputs in the “kongyiji” directory) are loaded. Include “kongyiji/input” and “kongyiji/prompts” as configuration and prompt resources.
- Config & Settings Management: Show the configuration layer that handles diverse input models, configuration files (located in graphrag/config and extra_data/settings.yaml) to control parameters for the LLM, caching, chunking, etc.
- Preprocessing and text splitting (graphrag/index/text_splitting)
- Graph construction and update (graphrag/index/graph, which covers embedding, extraction, clustering, layout, merge, and visualization modules)
- Workflows that drive the creation of various artifacts such as base documents, entities, relationships, and community reports (under graphrag/index/workflows/v1).
- Query Engine: Represent the querying subsystem (from graphrag/query). This separates into:
- Local Search (using internal vector stores and context builders)
- Global Search (likely using a combination of system prompts and external retrieval techniques)
- Vector Stores: Include a component for vector storage (graphrag/vector_stores) where embeddings are managed—this supports efficient similarity searches during queries.
- Visualization Interface: Illustrate the “visualize_graph” folder as a separate component. This uses notebooks and HTML artifacts (with caches of CSS and JS assets) to render interactive graphs via pyvis.
- Supporting Utilities & Caches: Represent supporting libraries for caching (graphrag/index/cache and caching mechanisms in the LLM layer), progress tracking, error handling, and other common utilities across the system.

##### Relationships and Data Flow
- Data flows from the Input Layer into preprocessing modules (text splitting and extraction). From there, the system applies LLM interactions for entity extraction and summarization.
- The processed data is then passed to the graph building pipeline where nodes (entities, text units, documents) and edges (relationships) are created.
- Once the graph is built and stored as artifacts (parquet files under kongyiji/output/*), the Query Engine (with its two methods: local and global) can access this data. Queries trigger context builders to interact with LLM module for retrieval.
- The Visualization Interface receives graph data from the stored artifacts and renders it interactively for user exploration.

##### Architectural Patterns & Design Principles
- Modular Pipelines: The codebase is organized as a set of decoupled modules (each under its own directory, e.g., extraction, embedding, workflows), following the principle of separation of concerns.
- Configuration-Driven Execution: The entire system’s behavior is highly determined by configuration files and YAML settings, making it extensible and customizable.
- LLM Abstraction: The LLM calls are wrapped in rate-limiting, history tracking, and caching layers to ensure robustness.
- Extensibility & Workflow Orchestration: Using workflow definitions (graphrag/index/workflows) allows clear control over different processing stages that can be independently updated or extended.

##### Relevant Technologies & Frameworks
- Python environment with dependencies listed in requirements.txt (with a note on uvloop exclusion on Windows).
- LLM integrations through OpenAI API clients and potentially Chinese LLM support (Qwen, Tongyi).
- Visualization using “pyvis” (with caching of external assets in visualize_graph/lib).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input & Data Ingestion Layer
    subgraph "Input Layer"
        DS["Data Source"]:::input
        IF["Input Files"]:::input
        PR["Prompt Resources"]:::input
    end

    %% Config & Settings Management Layer
    subgraph "Config Manager"
        CFG["System & LLM Config"]:::config
        ST1["General Settings"]:::config
        ST2["Project Config"]:::config
    end

    %% Core Processing / Indexing Engine
    subgraph "Processing Engine"
        PS["Text Preprocessor"]:::processing
        GC["Graph Constructor"]:::processing
        WP["Workflow Pipelines"]:::processing
    end

    %% Artifacts Storage (derived outputs)
    AL["Artifacts Storage"]:::storage

    %% LLM Integration & Abstraction
    subgraph "LLM Integration"
        LLM1["LLM Client"]:::llm
        LLM2["LLM Query Adapter"]:::llm
        LLM_CACHE["LLM Cache"]:::llm
    end

    %% Query Engine
    subgraph "Query Engine"
        QE["Query Engine"]:::query
    end

    %% Vector Stores
    VS["Vector Store"]:::storage

    %% Visualization Interface
    VIS["Visualization Module"]:::visual

    %% Supporting Utilities & Caches
    CACHE["Core Cache"]:::util

    %% Data Flow Connections
    DS -->|"ingests"| PS
    IF -->|"ingests"| PS
    PR -->|"ingests"| PS

    PS -->|"splits"| GC
    GC -->|"builds"| WP
    WP -->|"stores"| AL

    AL -->|"feeds"| QE
    AL -->|"renders"| VIS

    GC -->|"embeds"| VS
    QE -->|"retrieves"| VS

    PS -->|"callsLLM"| LLM1
    QE -->|"callsLLM"| LLM2

    LLM1 -.->|"cache_lookup"| LLM_CACHE

    %% Config flows
    ST1 -->|"feeds"| CFG
    ST2 -->|"feeds"| CFG
    CFG -->|"configures"| PS
    CFG -->|"configures"| GC
    CFG -->|"configures"| LLM1

    %% Supporting Cache connection
    CACHE -.-> PS

    %% Click Events
    click DS "https://github.com/yuezhengmeng/graphrag-learn/blob/master/data/kongyiji.txt"
    click IF "https://github.com/yuezhengmeng/graphrag-learn/tree/master/kongyiji/input"
    click PR "https://github.com/yuezhengmeng/graphrag-learn/tree/master/kongyiji/prompts"
    click CFG "https://github.com/yuezhengmeng/graphrag-learn/tree/master/graphrag/config"
    click ST1 "https://github.com/yuezhengmeng/graphrag-learn/blob/master/extra_data/settings.yaml"
    click ST2 "https://github.com/yuezhengmeng/graphrag-learn/blob/master/kongyiji/settings.yaml"
    click PS "https://github.com/yuezhengmeng/graphrag-learn/tree/master/graphrag/index/text_splitting"
    click GC "https://github.com/yuezhengmeng/graphrag-learn/tree/master/graphrag/index/graph"
    click WP "https://github.com/yuezhengmeng/graphrag-learn/tree/master/graphrag/index/workflows/v1"
    click LLM1 "https://github.com/yuezhengmeng/graphrag-learn/tree/master/graphrag/llm"
    click LLM2 "https://github.com/yuezhengmeng/graphrag-learn/tree/master/graphrag/query/llm"
    click QE "https://github.com/yuezhengmeng/graphrag-learn/tree/master/graphrag/query"
    click VS "https://github.com/yuezhengmeng/graphrag-learn/tree/master/graphrag/vector_stores"
    click VIS "https://github.com/yuezhengmeng/graphrag-learn/tree/master/visualize_graph"
    click CACHE "https://github.com/yuezhengmeng/graphrag-learn/tree/master/graphrag/index/cache"
    click LLM_CACHE "https://github.com/yuezhengmeng/graphrag-learn/tree/master/graphrag/llm/base"

    %% Styles
    classDef input fill:#DDEBF7,stroke:#333,stroke-width:2px;
    classDef config fill:#FFF2CC,stroke:#333,stroke-width:2px;
    classDef processing fill:#E2EFDA,stroke:#333,stroke-width:2px;
    classDef llm fill:#FCE4D6,stroke:#333,stroke-width:2px;
    classDef query fill:#F4CCCC,stroke:#333,stroke-width:2px;
    classDef storage fill:#D9E1F2,stroke:#333,stroke-width:2px;
    classDef visual fill:#EAD1DC,stroke:#333,stroke-width:2px;
    classDef util fill:#E6E6FA,stroke:#333,stroke-width:2px;
```
