How anmolwajid3/memory-agent-system works
1. What this repository is
- This project compares two answer-generation strategies for long-term conversational recall on LongMemEval-S:
- a full-context baseline that sends the whole conversation history to the model
- a structured memory system that extracts and retrieves only relevant facts
- The key architectural boundary is that each benchmark task is independent; the system does not carry memory across tasks, even though the UI presents them sequentially.
2. Top-level execution paths
- The main user entrypoint is gui/app.py, which launches a Streamlit app for running experiments and viewing results.
- The benchmark dataset is prepared via download_benchmark.py and the benchmark/ package.
- The experiment runtime flows through pipeline/runner.py, which coordinates agent execution and logging.
3. Benchmark/data layer
- benchmark/loader.py is responsible for loading the LongMemEval-S tasks and selecting a subset for experiments.
- This layer is the boundary between external dataset files and the internal agent pipeline.
- The repo’s README indicates the dataset is auto-downloaded on first run and the default experiment uses a 40-task subset, with a minimum configurable in the UI.
4. Pipeline orchestration layer
- The pipeline/ package is the control plane for each task.
- pipeline/graph.py and pipeline/state.py indicate the system is modeled as a state machine / LangGraph workflow.
- Supporting modules add operational concerns:
- retry.py for transient failure handling
- task_timeout.py for per-task execution limits
- token_counter.py for measuring model cost
- logger.py for structured experiment logs
- This layer is where task input, agent outputs, memory retrieval, evaluation, and final results are stitched together.
5. Agent subsystem
- The agents/ directory contains the main reasoning roles:
- task_agent.py: handles the task/question and likely the answer-generation flow
- memory_agent.py: manages memory lookup/update actions
- orchestrator.py: coordinates agent interactions
- evaluator_agent.py: checks answers against expected outcomes / scoring logic
- reflexion_agent.py: performs repair or self-correction when the first attempt is insufficient
- Architecturally, this is the intelligence layer above the pipeline state machine: the pipeline decides *when* agents run, and the agents decide *what* to do.
6. Memory subsystem
- The memory/ package implements the structured-memory condition.
- mem0_client.py connects to Mem0, which serves as the external memory backend.
- graph_store.py suggests memory is organized as a categorical graph rather than raw transcript chunks.
- consolidator.py likely extracts and merges facts into persistent memory entries.
- retriever.py retrieves relevant memory at query time.
- The main architectural idea here is a two-step memory cycle: store/consolidate facts after reading history, then retrieve only relevant facts when answering.
7. Comparison between the two experimental conditions
- The full-context system is the simplest path: it forwards the complete conversation history directly into the model.
- The memory system inserts a structured intermediate layer:
1. read the conversation history
2. extract salient facts
3. store them in the graph/Mem0-backed memory
4. retrieve only query-relevant context
5. generate the answer using that reduced context
- This comparison is the core experimental design and is reflected in the repo structure more than in a single monolithic application.
8. Evaluation and experiment logging
- The pipeline writes JSON experiment logs, which are meant to be consumed later rather than only displayed live.
- The analysis/analysis.py module turns those logs into plots/summary artifacts.
- This separates runtime execution from post-hoc analysis, which is important for reproducible benchmarking.
9. User interface
- gui/app.py provides a Streamlit dashboard for running the experiment and reviewing outcomes.
- The UI exposes:
- experiment execution
- a logs/results view
- graph generation after runs complete
- It is a presentation layer only; the substantive work is still done by the benchmark loader, pipeline, agents, and memory modules.
10. Configuration and operational dependencies
- config.py and .env.example centralize environment configuration, especially the OpenAI API key.
- The system targets Python 3.11+ and uses OpenAI GPT-4o-mini as the LLM runtime.
- The repo also relies on LangGraph for orchestration and Mem0 for memory persistence/retrieval, which materially shape the architecture by splitting reasoning from memory management.
11. Most important architecture boundaries to capture in a diagram
- UI vs execution engine: Streamlit triggers runs but does not contain the logic.
- Benchmark vs runtime: dataset loading is separate from task execution.
- Orchestration vs agents: the pipeline controls flow; agents implement behavior.
- Raw dialogue vs structured memory: the memory system transforms conversation into a smaller, queryable representation.
- Execution vs analysis: logs are produced during runs and analyzed afterward.
12. If you draw the system
- Represent the repo as four major zones:
- Input/Benchmark
- Orchestration/Pipeline
- Agent + Memory reasoning
- Logging/Analysis/UI
- Show two answer paths inside the runtime: full-context baseline and memory-backed retrieval, both ending in evaluation and logging.
Main entry points: runner.py Task Runner · orchestrator.py Orchestrator · graph.py Workflow Graph · memory_agent.py Memory Agent
Components 18
Input / Benchmark
Dataset acquisition and task loading for LongMemEval-S.
- Benchmark Loader · Loads LongMemEval-S tasks and selects experiment subsets.
benchmark/loader.py - Dataset Download · Fetches benchmark data on first run.
download_benchmark.py
Orchestration / Pipeline
Execution control plane for each independent task.
- Task Runner · Coordinates each task execution and logging.
pipeline/runner.py - Workflow Graph · Defines the state-machine workflow for a task.
pipeline/graph.py - Task State · Carries task input, intermediate outputs, and results.
pipeline/state.py - Experiment Logs · Writes JSON logs for later analysis.
pipeline/logger.py
Agents / Memory
Answer generation, memory operations, and recovery logic.
- Orchestrator · Coordinates the reasoning roles during a task.
agents/orchestrator.py - Task Agent · Handles question answering and final response generation.
agents/task_agent.py - Memory Agent · Manages memory lookup and update actions.
agents/memory_agent.py - Evaluator · Checks answers and produces task evaluation signals.
agents/evaluator_agent.py - Reflexion · Attempts self-correction after weak answers.
agents/reflexion_agent.py - Consolidator · Extracts and merges salient facts into memory.
memory/consolidator.py - Retriever · Fetches query-relevant structured memory.
memory/retriever.py - Graph Store · Organizes memory as a categorical graph.
memory/graph_store.py - Mem0 Client · Connects to external Mem0 persistence.
memory/mem0_client.py
Logging / Analysis / UI
Presentation, logs, and post-hoc analysis.
- Experiment UI · Runs experiments and shows results.
gui/app.py - Analysis · Turns run logs into plots and summaries.
analysis/analysis.py - Config · Central environment and API configuration.
config.py
Connections 21
- Experiment UI → Task Runner: start run · Streamlit triggers experiment execution.
- Experiment UI → Analysis: view results · UI presents post-run summaries and plots.
- Dataset Download → Benchmark Loader: bootstrap · Downloaded data is loaded through the benchmark layer.
- Benchmark Loader → Task Runner: tasks · Provides the selected LongMemEval-S tasks.
- Config → Experiment UI: settings · UI reads environment configuration.
- Config → Task Runner: llm/api · Runtime uses configured model and API credentials.
- Task Runner → Workflow Graph: orchestrate · Runner executes the task workflow.
- Workflow Graph → Task State: state · Workflow advances through task state transitions.
- Workflow Graph → Orchestrator: dispatch · Graph sends control to the agent coordinator.
- Orchestrator → Task Agent: answer · Coordinator invokes the task-solving agent.
- Orchestrator → Memory Agent: memory · Coordinator invokes memory operations when needed.
- Orchestrator → Evaluator: score · Coordinator asks for evaluation of outputs.
- Task Agent → Reflexion: repair · Weak answers can trigger self-correction.
- Memory Agent → Consolidator: store · Converts dialogue into structured facts.
- Consolidator → Graph Store: persist · Writes consolidated facts into graph memory.
- Graph Store → Mem0 Client: backend · Memory graph persists through Mem0.
- Memory Agent → Retriever: fetch · Retrieves relevant memory for the current query.
- Retriever → Graph Store: query · Retrieval reads from the structured memory store.
- Evaluator → Experiment Logs: emit · Evaluation outcomes are written to logs.
- Task Runner → Experiment Logs: record · Runner records execution events and metrics.
- Experiment Logs → Analysis: consume · Analysis reads experiment logs after runs complete.