How xiaoxinhe/g-retriever works
Project Purpose and Type
• This is a research-oriented full-stack project for “Retrieval-Augmented Generation” that integrates Graph Neural Networks (GNNs), Large Language Models (LLMs) and Retrieval-Augmentation techniques.
• Its primary use cases include textual graph understanding and question answering over datasets like scene graphs, expla_graphs, GQA, and WebQSP.
Main Components
a. Data Layer:
• Preprocessing: Represent the “src/dataset/preprocess” modules (e.g., expla_graphs.py, scene_graphs.py, webqsp.py, generate_split.py) as a component that cleans and splits raw data.
b. Model Layer:
• Graph Modules: Illustrate the “Graph Neural Network” component (from src/model/gnn.py) that processes graphical representations.
• LLM Components: Show components for the large language models. The files (llm.py, pt_llm.py) and the specialized “graph_llm” (for G-Retriever) indicate different model training scenarios:
– Inference-only LLM
– Frozen LLM with Prompt Tuning (pt_llm)
– Tuned LLM / G-Retriever with LoRA
These should be depicted as alternative pipelines or branches.
c. Integration Module:
d. Utility and Supporting Components:
• Configuration and Utilities: Include “src/config.py” and utility modules such as evaluation, checkpoint management, scheduler (e.g., lr_schedule.py), and collate functions.
• Execution Scripts: Represent the training pipeline (train.py) and inference workflow (inference.py) as orchestrators that glue together pre-processing, model training, and evaluation.
• External Dependencies: Optionally note critical external technologies like PyTorch, PyTorch Geometric, Transformers, and additional libraries (e.g., wandb, datasets) which are core parts of the environment setup.
Relationships and Interaction Flows
• Processed data then should flow into the Training Pipeline (train.py) as well as the Inference Pipeline (inference.py), triggering separate model interactions.
• Within the model training block, branch it into two or three sub-components:
– One branch for the basic LLM (inference-only or fine-tuned LLM),
– One for prompt tuning using the frozen LLM,
– And one for the G-Retriever model that integrates GNN and LLM.
Architectural Patterns/Design Principles
• Hybrid Model Integration: Emphasize the dual nature of the approach—integrating graph-based learning (GNN) and natural language understanding (LLM) tuned via retrieval augmented methods.
• Flexibility: Indicate how various training modes (inference-only, prompt tuning, LoRA fine-tuning) dynamically connect to the same core preprocessed data.
• Annotations for each training/inference type (inference-only vs. prompt-tuning vs. full fine-tuning), showing variations in the model architecture.