# xiaowu0162/longmemeval architecture

> Architecture diagram of the GitHub repository xiaowu0162/longmemeval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xiaowu0162/longmemeval
- Repository: https://github.com/xiaowu0162/longmemeval

## Overview

##### Project Type and Purpose
- The project is a full experimental benchmark framework aimed at testing and improving long-term memory retrieval and question answering in conversational AI systems.
- Its purpose is to compile, manage, and evaluate large, timestamped chat histories (datasets) to test different memory abilities (e.g., temporal reasoning, multi-session recall). It supports full memory system experiments and custom history generation.

##### Analyzing the File Structure
- Top-level directories include:
- assets: contains static files like images (used for documentation/visuals).
- data: holds the various JSON files that are the evaluation datasets (including a custom history sample).
- src: the core source code is organized into components:
∙ evaluation – scripts for computing QA metrics and retrieval evaluation.
∙ generation – code for long-context generation experiments (including shell scripts for running models).
∙ index_expansion – code to expand indexes (e.g., keyphrases or session-level summaries) to improve retrieval.
∙ retrieval – modules and scripts for memory retrieval experiments using various retrievers.
∙ utils – utility scripts for serving models locally (e.g., with vllm).
- There are also requirements files and shell scripts for setting up the environment, showing that the project supports both “evaluation only” and “full support” modes.

##### Components and Their Interactions
a. Data and Dataset Management:
- The data directory contains the evaluation instances (longmemeval_s.json, longmemeval_m.json, longmemeval_oracle.json).
- There is support for custom history generation (under data/custom_history and the sample script file).
b. Evaluation Module:
- Located in src/evaluation, these scripts (evaluate_qa.py, print_qa_metrics.py, print_retrieval_metrics.py) handle metric computation against provided predicted answers.
c. Generation Module:
- In src/generation, you have scripts for running long-context generation on chat histories, including shell scripts (run_generation.sh) to interface either with OpenAI models or with vllm servers.
d. Retrieval Module:
- Located in src/retrieval, it supports multiple retrieval strategies (BM25, dense retrievers like contriever, Stella, etc.) configured via shell scripts (run_retrieval.sh) as well as utilities to compute or print evaluation metrics.
e. Index Expansion:
- Found in src/index_expansion, these scripts are used to run various forms of expansion (session summary, keyphrase extraction, user facts, etc.) to enhance the memory indexing process.
f. Utility & Infrastructure Support:
- The utils subdirectory supports model serving (e.g., serve_vllm.sh) which is integral for running an open-weight reader model locally.

##### Architectural Patterns and Relationships
- Data ingestion/management (data directory) feeds into evaluation and generation experiments.
- Retrieval and index expansion modules interact to refine the memory search (using both original and expanded keys).
- The evaluation modules consume the outputs from both retrieval experiments and generation experiments to calculate performance metrics.
- For example: “Data (chat histories) → Retrieval Module → Retrieval Results → Generation Module” and “Output Answers → Evaluation Module → Metrics Calculation.”
- That the system supports both retrieval-augmented generation and raw long-context generation experiments.

##### Technologies and Frameworks
- The project relies on Python (with separate requirements for lite and full experiments), conda for environment management, and Torch (for model inference).
- It supports both API-based calls (e.g., OpenAI models) and locally served models using vllm.
- Shell scripts orchestrate the experiments, indicating a pipeline-like execution model.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        ds["Data Source (JSON Datasets\n& Custom History)"]:::data
    end

    subgraph "Experimental Pipelines"
        retrieval["Retrieval Engine\n(Various retrievers)"]:::retrieval
        indexExpansion["Index Expansion Module\n(Enhances retrieval)"]:::index
        generation["Generation Engine\n(API & VLLM Generation)"]:::generation
    end

    subgraph "Evaluation Layer"
        evaluation["Evaluation Engine\n(QA & Retrieval Metrics)"]:::evaluation
    end

    subgraph "Utility Layer"
        utility["Utility/Infrastructure\n(Model Serving via VLLM)"]:::utility
    end

    %% Data Flow
    ds -->|"feeds"| retrieval
    ds -->|"feeds"| generation
    indexExpansion -->|"augments"| retrieval
    retrieval -->|"provides results to"| generation
    generation -->|"outputs answers to"| evaluation
    retrieval -->|"reports metrics to"| evaluation
    utility -->|"supports"| generation

    %% Click Events
    click ds "https://github.com/xiaowu0162/longmemeval/tree/main/data"
    click evaluation "https://github.com/xiaowu0162/longmemeval/tree/main/src/evaluation"
    click generation "https://github.com/xiaowu0162/longmemeval/tree/main/src/generation"
    click retrieval "https://github.com/xiaowu0162/longmemeval/tree/main/src/retrieval"
    click indexExpansion "https://github.com/xiaowu0162/longmemeval/tree/main/src/index_expansion"
    click utility "https://github.com/xiaowu0162/longmemeval/tree/main/src/utils"

    %% Styles
    classDef data fill:#AED581,stroke:#33691E,stroke-width:2px;
    classDef evaluation fill:#FF8A65,stroke:#BF360C,stroke-width:2px;
    classDef generation fill:#CE93D8,stroke:#6A1B9A,stroke-width:2px;
    classDef retrieval fill:#4FC3F7,stroke:#01579B,stroke-width:2px;
    classDef index fill:#FFF176,stroke:#F57F17,stroke-width:2px;
    classDef utility fill:#A1887F,stroke:#4E342E,stroke-width:2px;
```
