How xiaowu0162/longmemeval works
Project Type and Purpose
• The project is a full experimental benchmark framework aimed at testing and improving long-term memory retrieval and question answering in conversational AI systems.
• Its purpose is to compile, manage, and evaluate large, timestamped chat histories (datasets) to test different memory abilities (e.g., temporal reasoning, multi-session recall). It supports full memory system experiments and custom history generation.
Analyzing the File Structure
• Top-level directories include:
• assets: contains static files like images (used for documentation/visuals).
• data: holds the various JSON files that are the evaluation datasets (including a custom history sample).
• src: the core source code is organized into components:
∙ evaluation – scripts for computing QA metrics and retrieval evaluation.
∙ generation – code for long-context generation experiments (including shell scripts for running models).
∙ index_expansion – code to expand indexes (e.g., keyphrases or session-level summaries) to improve retrieval.
∙ retrieval – modules and scripts for memory retrieval experiments using various retrievers.
∙ utils – utility scripts for serving models locally (e.g., with vllm).
• There are also requirements files and shell scripts for setting up the environment, showing that the project supports both “evaluation only” and “full support” modes.
Components and Their Interactions
a. Data and Dataset Management:
• The data directory contains the evaluation instances (longmemeval_s.json, longmemeval_m.json, longmemeval_oracle.json).
• There is support for custom history generation (under data/custom_history and the sample script file).
b. Evaluation Module:
• Located in src/evaluation, these scripts (evaluate_qa.py, print_qa_metrics.py, print_retrieval_metrics.py) handle metric computation against provided predicted answers.
c. Generation Module:
• In src/generation, you have scripts for running long-context generation on chat histories, including shell scripts (run_generation.sh) to interface either with OpenAI models or with vllm servers.
d. Retrieval Module:
• Located in src/retrieval, it supports multiple retrieval strategies (BM25, dense retrievers like contriever, Stella, etc.) configured via shell scripts (run_retrieval.sh) as well as utilities to compute or print evaluation metrics.
e. Index Expansion:
• Found in src/index_expansion, these scripts are used to run various forms of expansion (session summary, keyphrase extraction, user facts, etc.) to enhance the memory indexing process.
f. Utility & Infrastructure Support:
• The utils subdirectory supports model serving (e.g., serve_vllm.sh) which is integral for running an open-weight reader model locally.
Architectural Patterns and Relationships
• Data ingestion/management (data directory) feeds into evaluation and generation experiments.
• Retrieval and index expansion modules interact to refine the memory search (using both original and expanded keys).
• The evaluation modules consume the outputs from both retrieval experiments and generation experiments to calculate performance metrics.
• For example: “Data (chat histories) → Retrieval Module → Retrieval Results → Generation Module” and “Output Answers → Evaluation Module → Metrics Calculation.”
• That the system supports both retrieval-augmented generation and raw long-context generation experiments.
Technologies and Frameworks
• The project relies on Python (with separate requirements for lite and full experiments), conda for environment management, and Torch (for model inference).
• It supports both API-based calls (e.g., OpenAI models) and locally served models using vllm.
• Shell scripts orchestrate the experiments, indicating a pipeline-like execution model.