How rzhub/gatemem works
GateMem is a benchmark toolkit for testing memory governance in multi-principal LLM agents. Researchers run a baseline against domain episodes and checkpoints that probe utility, access control, and active forgetting; the evaluator scores predictions using governance metrics. The repository also supports generating prediction files for external submissions and leaderboard review. The README establishes this overall workflow and the major baseline families; source excerpts are limited, so internal call wiring beyond documented evaluation flow is kept conservative.
Main entry points: agents Baseline Agents · run_eval.py Evaluation CLI · runner.py Evaluation Runner · retrieval Retrieval Methods
Components 20
Benchmark Data
- Domain Episodes
bench/data - Domain Definitions
bench/domains.py
Agent Methods
- Baseline Agents
bench/agents - Retrieval Methods
bench/retrieval - Mem0 Baseline
bench/mem0 - ReMeM Baseline
bench/remem - Upstream Mem0
bench/integrations/mem0_upstream
Model Runtime
- LLM Providers
bench/llm/router.py - Embedding Providers
bench/embeddings/router.py
Evaluation
- Evaluation CLI
bench/scripts/run_eval.py - Evaluation Runner
bench/eval/runner.py - Prediction Validation
bench/eval/validator.py - LLM Judge
bench/eval/judge.py - Governance Metrics
bench/eval/metrics.py - Predictions JSONL
Other components
- Researcher
- LLM Service
- Embedding Service
- Submission Service
- Public Leaderboard
Connections 20
- Researcher → Evaluation CLI: runs benchmark
- Evaluation CLI → Domain Episodes: loads episodes
- Evaluation CLI → Domain Definitions: selects domain
- Evaluation CLI → Baseline Agents: selects baseline
- Baseline Agents → LLM Providers: requests inference
- LLM Providers → LLM Service: calls model
- Baseline Agents → Retrieval Methods: uses retrieval
- Retrieval Methods → Embedding Providers: requests embeddings
- Embedding Providers → Embedding Service: calls provider
- Baseline Agents → Mem0 Baseline: selects method
- Baseline Agents → ReMeM Baseline: selects method
- Mem0 Baseline → Upstream Mem0: optional backend
- Evaluation CLI → Evaluation Runner: evaluates run
- Evaluation Runner → Prediction Validation: validates output
- Evaluation Runner → LLM Judge: requests judging
- Evaluation Runner → Governance Metrics: computes scores
- Evaluation Runner → Predictions JSONL: writes predictions
- Researcher → Submission Service: uploads predictions
- Submission Service → Predictions JSONL: scores file
- Submission Service → Public Leaderboard: approved results