# rzhub/gatemem architecture

> Architecture diagram of the GitHub repository rzhub/gatemem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rzhub/gatemem
- Repository: https://github.com/rzhub/gatemem
- Video tour (about a minute): https://gitdiagram.com/rzhub/gatemem/video

## Overview

GateMem is a benchmark toolkit for testing memory governance in multi-principal LLM agents. Researchers run a baseline against domain episodes and checkpoints that probe utility, access control, and active forgetting; the evaluator scores predictions using governance metrics. The repository also supports generating prediction files for external submissions and leaderboard review. The README establishes this overall workflow and the major baseline families; source excerpts are limited, so internal call wiring beyond documented evaluation flow is kept conservative.

Main entry points: [`bench/agents`](https://github.com/rzhub/gatemem/tree/main/bench/agents) (Baseline Agents), [`bench/scripts/run_eval.py`](https://github.com/rzhub/gatemem/blob/main/bench/scripts/run_eval.py) (Evaluation CLI), [`bench/eval/runner.py`](https://github.com/rzhub/gatemem/blob/main/bench/eval/runner.py) (Evaluation Runner), [`bench/retrieval`](https://github.com/rzhub/gatemem/tree/main/bench/retrieval) (Retrieval Methods)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_benchmark["Benchmark Data"]
  node_domain_data["Domain Episodes"]
  node_domain_specs["Domain Definitions<br/>[domains.py]"]
end

subgraph group_agents["Agent Methods"]
  node_agent_methods["Baseline Agents"]
  node_retrieval["Retrieval Methods"]
  node_mem0["Mem0 Baseline"]
  node_remem["ReMeM Baseline"]
  node_upstream_mem0["Upstream Mem0"]
end

subgraph group_runtime["Model Runtime"]
  node_llm_router["LLM Providers<br/>[router.py]"]
  node_embedding_router["Embedding Providers<br/>[router.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_run_eval["Evaluation CLI<br/>[run_eval.py]"]
  node_eval_runner["Evaluation Runner<br/>[runner.py]"]
  node_validator["Prediction Validation<br/>[validator.py]"]
  node_judge["LLM Judge<br/>[judge.py]"]
  node_metrics["Governance Metrics<br/>[metrics.py]"]
  node_prediction_file["Predictions JSONL"]
end

node_researcher(("Researcher"))
node_llm_service(("LLM Service"))
node_embedding_service(("Embedding Service"))
node_submission(("Submission Service"))
node_leaderboard(("Public Leaderboard"))

node_researcher -->|"runs benchmark"| node_run_eval
node_run_eval -->|"loads episodes"| node_domain_data
node_run_eval -->|"selects domain"| node_domain_specs
node_run_eval -->|"selects baseline"| node_agent_methods
node_agent_methods -->|"requests inference"| node_llm_router
node_llm_router -.->|"calls model"| node_llm_service
node_agent_methods -.->|"uses retrieval"| node_retrieval
node_retrieval -->|"requests embeddings"| node_embedding_router
node_embedding_router -.->|"calls provider"| node_embedding_service
node_agent_methods -->|"selects method"| node_mem0
node_agent_methods -->|"selects method"| node_remem
node_mem0 -.->|"optional backend"| node_upstream_mem0
node_run_eval -->|"evaluates run"| node_eval_runner
node_eval_runner -->|"validates output"| node_validator
node_eval_runner -.->|"requests judging"| node_judge
node_eval_runner -->|"computes scores"| node_metrics
node_eval_runner -->|"writes predictions"| node_prediction_file
node_researcher -->|"uploads predictions"| node_submission
node_submission -->|"scores file"| node_prediction_file
node_submission -.->|"approved results"| node_leaderboard

click node_run_eval "https://github.com/rzhub/gatemem/blob/main/bench/scripts/run_eval.py"
click node_domain_data "https://github.com/rzhub/gatemem/tree/main/bench/data"
click node_domain_specs "https://github.com/rzhub/gatemem/blob/main/bench/domains.py"
click node_agent_methods "https://github.com/rzhub/gatemem/tree/main/bench/agents"
click node_llm_router "https://github.com/rzhub/gatemem/blob/main/bench/llm/router.py"
click node_embedding_router "https://github.com/rzhub/gatemem/blob/main/bench/embeddings/router.py"
click node_retrieval "https://github.com/rzhub/gatemem/tree/main/bench/retrieval"
click node_mem0 "https://github.com/rzhub/gatemem/tree/main/bench/mem0"
click node_remem "https://github.com/rzhub/gatemem/tree/main/bench/remem"
click node_upstream_mem0 "https://github.com/rzhub/gatemem/tree/main/bench/integrations/mem0_upstream"
click node_eval_runner "https://github.com/rzhub/gatemem/blob/main/bench/eval/runner.py"
click node_validator "https://github.com/rzhub/gatemem/blob/main/bench/eval/validator.py"
click node_judge "https://github.com/rzhub/gatemem/blob/main/bench/eval/judge.py"
click node_metrics "https://github.com/rzhub/gatemem/blob/main/bench/eval/metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_domain_data,node_domain_specs toneBlue
class node_agent_methods,node_retrieval,node_mem0,node_remem,node_upstream_mem0 toneAmber
class node_llm_router,node_embedding_router toneMint
class node_run_eval,node_eval_runner,node_validator,node_judge,node_metrics,node_prediction_file toneRose
class node_researcher,node_llm_service,node_embedding_service,node_submission,node_leaderboard toneIndigo
```

## Components

### Benchmark Data

- [Domain Episodes](https://github.com/rzhub/gatemem/tree/main/bench/data): `bench/data`
- [Domain Definitions](https://github.com/rzhub/gatemem/blob/main/bench/domains.py): `bench/domains.py`

### Agent Methods

- [Baseline Agents](https://github.com/rzhub/gatemem/tree/main/bench/agents): `bench/agents`
- [Retrieval Methods](https://github.com/rzhub/gatemem/tree/main/bench/retrieval): `bench/retrieval`
- [Mem0 Baseline](https://github.com/rzhub/gatemem/tree/main/bench/mem0): `bench/mem0`
- [ReMeM Baseline](https://github.com/rzhub/gatemem/tree/main/bench/remem): `bench/remem`
- [Upstream Mem0](https://github.com/rzhub/gatemem/tree/main/bench/integrations/mem0_upstream): `bench/integrations/mem0_upstream`

### Model Runtime

- [LLM Providers](https://github.com/rzhub/gatemem/blob/main/bench/llm/router.py): `bench/llm/router.py`
- [Embedding Providers](https://github.com/rzhub/gatemem/blob/main/bench/embeddings/router.py): `bench/embeddings/router.py`

### Evaluation

- [Evaluation CLI](https://github.com/rzhub/gatemem/blob/main/bench/scripts/run_eval.py): `bench/scripts/run_eval.py`
- [Evaluation Runner](https://github.com/rzhub/gatemem/blob/main/bench/eval/runner.py): `bench/eval/runner.py`
- [Prediction Validation](https://github.com/rzhub/gatemem/blob/main/bench/eval/validator.py): `bench/eval/validator.py`
- [LLM Judge](https://github.com/rzhub/gatemem/blob/main/bench/eval/judge.py): `bench/eval/judge.py`
- [Governance Metrics](https://github.com/rzhub/gatemem/blob/main/bench/eval/metrics.py): `bench/eval/metrics.py`
- **Predictions JSONL**

### Other components

- **Researcher**
- **LLM Service**
- **Embedding Service**
- **Submission Service**
- **Public Leaderboard**

## Connections

- Researcher → Evaluation CLI: runs benchmark
- Evaluation CLI → Domain Episodes: loads episodes
- Evaluation CLI → Domain Definitions: selects domain
- Evaluation CLI → Baseline Agents: selects baseline
- Baseline Agents → LLM Providers: requests inference
- LLM Providers → LLM Service: calls model
- Baseline Agents → Retrieval Methods: uses retrieval
- Retrieval Methods → Embedding Providers: requests embeddings
- Embedding Providers → Embedding Service: calls provider
- Baseline Agents → Mem0 Baseline: selects method
- Baseline Agents → ReMeM Baseline: selects method
- Mem0 Baseline → Upstream Mem0: optional backend
- Evaluation CLI → Evaluation Runner: evaluates run
- Evaluation Runner → Prediction Validation: validates output
- Evaluation Runner → LLM Judge: requests judging
- Evaluation Runner → Governance Metrics: computes scores
- Evaluation Runner → Predictions JSONL: writes predictions
- Researcher → Submission Service: uploads predictions
- Submission Service → Predictions JSONL: scores file
- Submission Service → Public Leaderboard: approved results