# xiaoxinhe/g-retriever architecture

> Architecture diagram of the GitHub repository xiaoxinhe/g-retriever, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xiaoxinhe/g-retriever
- Repository: https://github.com/xiaoxinhe/g-retriever

## Overview

##### Project Purpose and Type
- This is a research-oriented full-stack project for “Retrieval-Augmented Generation” that integrates Graph Neural Networks (GNNs), Large Language Models (LLMs) and Retrieval-Augmentation techniques.
- Its primary use cases include textual graph understanding and question answering over datasets like scene graphs, expla_graphs, GQA, and WebQSP.

##### Main Components
a. Data Layer:
- Preprocessing: Represent the “src/dataset/preprocess” modules (e.g., expla_graphs.py, scene_graphs.py, webqsp.py, generate_split.py) as a component that cleans and splits raw data.

b. Model Layer:
- Graph Modules: Illustrate the “Graph Neural Network” component (from src/model/gnn.py) that processes graphical representations.
- LLM Components: Show components for the large language models. The files (llm.py, pt_llm.py) and the specialized “graph_llm” (for G-Retriever) indicate different model training scenarios:
– Inference-only LLM
– Frozen LLM with Prompt Tuning (pt_llm)
– Tuned LLM / G-Retriever with LoRA
These should be depicted as alternative pipelines or branches.

c. Integration Module:

d. Utility and Supporting Components:
- Configuration and Utilities: Include “src/config.py” and utility modules such as evaluation, checkpoint management, scheduler (e.g., lr_schedule.py), and collate functions.
- Execution Scripts: Represent the training pipeline (train.py) and inference workflow (inference.py) as orchestrators that glue together pre-processing, model training, and evaluation.
- External Dependencies: Optionally note critical external technologies like PyTorch, PyTorch Geometric, Transformers, and additional libraries (e.g., wandb, datasets) which are core parts of the environment setup.

##### Relationships and Interaction Flows
- Processed data then should flow into the Training Pipeline (train.py) as well as the Inference Pipeline (inference.py), triggering separate model interactions.
- Within the model training block, branch it into two or three sub-components:
– One branch for the basic LLM (inference-only or fine-tuned LLM),
– One for prompt tuning using the frozen LLM,
– And one for the G-Retriever model that integrates GNN and LLM.

##### Architectural Patterns/Design Principles
- Hybrid Model Integration: Emphasize the dual nature of the approach—integrating graph-based learning (GNN) and natural language understanding (LLM) tuned via retrieval augmented methods.
- Flexibility: Indicate how various training modes (inference-only, prompt tuning, LoRA fine-tuning) dynamically connect to the same core preprocessed data.

- Annotations for each training/inference type (inference-only vs. prompt-tuning vs. full fine-tuning), showing variations in the model architecture.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        DS1("Expla Graphs"):::data
        DS2("Scene Graphs"):::data
        DS3("GQA"):::data
        DP("Data Preprocessing"):::processing
        DS1 --> DP
        DS2 --> DP
        DS3 --> DP
    end

    %% Execution Layer
    subgraph "Execution Layer"
        TP("Training Pipeline (train.py)"):::execution
        IP("Inference Workflow (inference.py)"):::execution
        RS("Run Script (run.sh)"):::execution
        DP --> TP
        DP --> IP
        RS --> TP
        RS --> IP
    end

    %% Model Layer
    subgraph "Model Layer"
        GNN("GNN Module"):::model
        LLM1("LLM Inference-only Module"):::model
        LLM2("LLM Frozen Prompt Tuning"):::model
        GR("G-Retriever"):::model
        TP --> GNN
        TP --> LLM1
        TP --> LLM2
        TP --> GR
        IP --> GNN
        IP --> LLM1
        IP --> LLM2
        IP --> GR
    end

    %% Integration
    subgraph "Integration"
        RAG("RAG Integration"):::integration
        GNN --> RAG
        LLM1 --> RAG
        LLM2 --> RAG
        GR --> RAG
        RAG --> OE("Output/Evaluation"):::processing
    end

    %% Utilities & Configuration
    subgraph "Utilities & Configuration"
        CFG("Configuration"):::utility
        CKPT("Checkpoint Management"):::utility
        SCH("Scheduler"):::utility
        COL("Collate Functions"):::utility
        EV("Evaluation"):::utility
        OU("Other Utilities"):::utility
        TP --> CFG
        TP --> CKPT
        TP --> SCH
        TP --> COL
        TP --> EV
        TP --> OU
        IP --> CFG
        IP --> CKPT
        IP --> SCH
        IP --> COL
        IP --> EV
        IP --> OU
    end

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef processing fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef model fill:#D8BFD8,stroke:#000,stroke-width:1px;
    classDef integration fill:#FFE4B5,stroke:#000,stroke-width:1px;
    classDef execution fill:#FFD700,stroke:#000,stroke-width:1px;
    classDef utility fill:#F5F5DC,stroke:#000,stroke-width:1px;

    %% Click Events
    click DS1 "https://github.com/xiaoxinhe/g-retriever/tree/main/dataset/expla_graphs"
    click DS2 "https://github.com/xiaoxinhe/g-retriever/tree/main/dataset/scene_graphs"
    click DS3 "https://github.com/xiaoxinhe/g-retriever/tree/main/dataset/gqa"
    click DP "https://github.com/xiaoxinhe/g-retriever/tree/main/src/dataset/preprocess"
    click GNN "https://github.com/xiaoxinhe/g-retriever/blob/main/src/model/gnn.py"
    click LLM1 "https://github.com/xiaoxinhe/g-retriever/blob/main/src/model/llm.py"
    click LLM2 "https://github.com/xiaoxinhe/g-retriever/blob/main/src/model/pt_llm.py"
    click GR "https://github.com/xiaoxinhe/g-retriever/blob/main/src/model/graph_llm.py"
    click RAG "https://github.com/xiaoxinhe/g-retriever/blob/main/src/dataset/utils/retrieval.py"
    click CFG "https://github.com/xiaoxinhe/g-retriever/blob/main/src/config.py"
    click CKPT "https://github.com/xiaoxinhe/g-retriever/blob/main/src/utils/ckpt.py"
    click SCH "https://github.com/xiaoxinhe/g-retriever/blob/main/src/utils/lr_schedule.py"
    click COL "https://github.com/xiaoxinhe/g-retriever/blob/main/src/utils/collate.py"
    click EV "https://github.com/xiaoxinhe/g-retriever/blob/main/src/utils/evaluate.py"
    click OU "https://github.com/xiaoxinhe/g-retriever/blob/main/src/utils/lm_modeling.py"
    click TP "https://github.com/xiaoxinhe/g-retriever/blob/main/train.py"
    click IP "https://github.com/xiaoxinhe/g-retriever/blob/main/inference.py"
    click RS "https://github.com/xiaoxinhe/g-retriever/blob/main/run.sh"
```
