# agent-on-the-fly/memento architecture

> Architecture diagram of the GitHub repository agent-on-the-fly/memento, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/agent-on-the-fly/memento
- Repository: https://github.com/agent-on-the-fly/memento

## Overview

##### Project Type and Purpose
- This is a modular, agent-based system that implements a two-stage planner–executor architecture on top of large language models, augmented with a memory bank for continual learning. It is not a traditional full-stack web app but a distributed client/server framework:
- A lightweight “client” (the Planner–Executor agent)
- A set of “server” MCP (Multi-Context Protocol) tool servers
- A memory component for experience replay
- Integrations with external services (OpenAI LLMs, SearxNG, FFmpeg, SerpAPI, etc.)

##### File Structure Analysis
- Top-level directories:
- client/ – the agent entrypoint and CBR logic (Planner + Executor)
- server/ – individual MCP tool servers (code, search, document, image, video, etc.)
- interpreters/ – backends for sandboxed code execution (Docker, subprocess, IPython, internal Python)
- memory/ – memory implementations and case-bank data store
- data/ – sample input data and cases
- searxng-docker/ – configuration for SearxNG search engine
- Figure/, README.md, requirements, LICENSE, pyproject.toml

##### Key Components
a. Client Agent
- Meta-Planner module (task decomposition via GPT-4.1)
- Executor module (calls MCP tools)
- Case-Based Reasoning (CBR) layer (no-parametric memory)
- Memory adapter for Read/Write operations to Case Bank
b. MCP Tool Layer (server/)
- Web tools: search_tool, serp_search, ai_crawler, craw_page
- Data tools: documents_tool, excel_tool, math_tool
- Media tools: image_tool, video_tool
- Code tool: code_agent + interpreters
c. Interpreters
- BaseInterpreter + Docker, subprocess, IPython, internal Python, E2B
d. Memory
- Case Bank (memory/np_memory.py or dummy_memo.jsonl)
- Data store (data/deepresearcher.jsonl)
e. External Services
- OpenAI API (for planner & executor models)
- SearxNG search (via searxng-docker)
- FFmpeg (system binary for video)
- Optional APIs: Chunkr, Jina, AssemblyAI, SerpAPI

##### Relationships and Data Flows
- User Input → Client Agent (agent.py)
- Meta-Planner → OpenAI GPT endpoint → subtasks list
- Planner reads cases from Case Bank → refines subtask ordering
- Executor invokes MCP calls over HTTP/GRPC to server tools
- Server tools may spin up interpreters (for code), call SearxNG (search), FFmpeg (video), or external REST APIs
- Tool results → returned to Executor → aggregator → final response
- After each subtask or trajectory, the agent writes experience tuples to Case Bank
- Memory queries influence future Planner decisions

##### Architectural Patterns & Principles
- Decoupled Planner vs. Executor
- Memory-Augmented MDP modeled as CBR policy
- Microservice-style MCP servers for extensibility
- Interpreter abstraction for secure, sandboxed code execution
- Externalization of search engine via Docker (SearxNG)

##### Final Checklist
- All major components boxed and labeled
- Arrowheads indicating direction of calls and data flow
- Annotations for protocols (HTTP, MCP) and technologies
- A clear title: “Memento System Architecture”

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Agent Cluster
    subgraph "Client Agent"
        Planner["Meta-Planner\n(GPT-4.1)"]:::core
        Executor["Executor\n(uv + FastMCP)"]:::core
        CBR["CBR Layer"]:::core
        MemAdapter["Memory Adapter"]:::core
    end

    %% Memory Cluster
    subgraph "Memory"
        CaseBank["Case Bank\n(data store)"]:::memory
        DummyStore["Dummy Memory Store"]:::memory
    end

    %% MCP Tools Cluster
    subgraph "MCP Tools"
        subgraph "Web Tools"
            SearchTool["Web Search Tool"]:::tool
            SerpSearch["SerpAPI Search"]:::tool
            AICrawl["AI Crawler"]:::tool
            PageCrawl["Page Crawler"]:::tool
        end
        subgraph "Data Tools"
            DocTool["Documents Tool"]:::tool
            ExcelTool["Excel Tool"]:::tool
            MathTool["Math Tool"]:::tool
        end
        subgraph "Media Tools"
            ImageTool["Image Tool"]:::tool
            VideoTool["Video Tool"]:::tool
        end
        CodeAgent["Code Execution Agent"]:::tool
    end

    %% Interpreters Cluster
    subgraph "Interpreters"
        BaseInterp["BaseInterpreter"]:::interp
        DockerInterp["Docker Interpreter"]:::interp
        SubpInterp["Subprocess Interpreter"]:::interp
        IPythonInterp["IPython Interpreter"]:::interp
        InternalInterp["Internal Python Interpreter"]:::interp
        E2BInterp["E2B Interpreter"]:::interp
    end

    %% External Services Cluster
    subgraph "External Services"
        OpenAI["OpenAI API"]:::external
        SearxNG["SearxNG (Docker)"]:::external
        FFmpeg["FFmpeg"]:::external
        SerpAPI["SerpAPI"]:::external
        AssemblyAI["AssemblyAI"]:::external
        Jina["Jina"]:::external
        Chunkr["Chunkr"]:::external
    end

    %% User Input
    User["User"]:::external

    %% Data Flows
    User -->|Input| Planner
    Planner -->|LLM call| OpenAI
    Planner -->|case read| CaseBank
    Planner -->|case read| DummyStore
    Planner -->|subtasks| Executor

    Executor -->|"HTTP MCP"| SearchTool
    Executor -->|"HTTP MCP"| SerpSearch
    Executor -->|"HTTP MCP"| AICrawl
    Executor -->|"HTTP MCP"| PageCrawl
    Executor -->|"HTTP MCP"| DocTool
    Executor -->|"HTTP MCP"| ExcelTool
    Executor -->|"HTTP MCP"| MathTool
    Executor -->|"HTTP MCP"| ImageTool
    Executor -->|"HTTP MCP"| VideoTool
    Executor -->|"HTTP MCP"| CodeAgent

    CodeAgent -->|spawns| BaseInterp
    BaseInterp -->|loads| DockerInterp
    BaseInterp -->|loads| SubpInterp
    BaseInterp -->|loads| IPythonInterp
    BaseInterp -->|loads| InternalInterp
    BaseInterp -->|loads| E2BInterp

    SearchTool -->|calls| SearxNG
    SerpSearch -->|calls| SerpAPI
    VideoTool -->|calls| FFmpeg

    Executor -->|write case| CaseBank
    CaseBank -->|feedback| Planner

    %% Click Events
    click Planner "https://github.com/agent-on-the-fly/memento/blob/main/client/agent.py"
    click Executor "https://github.com/agent-on-the-fly/memento/blob/main/client/agent.py"
    click CBR "https://github.com/agent-on-the-fly/memento/blob/main/client/no_parametric_cbr.py"
    click MemAdapter "https://github.com/agent-on-the-fly/memento/blob/main/memory/np_memory.py"
    click DummyStore "https://github.com/agent-on-the-fly/memento/blob/main/memory/dummy_memo.jsonl"
    click CaseBank "https://github.com/agent-on-the-fly/memento/blob/main/data/deepresearcher.jsonl"
    click SearchTool "https://github.com/agent-on-the-fly/memento/blob/main/server/search_tool.py"
    click SerpSearch "https://github.com/agent-on-the-fly/memento/blob/main/server/serp_search.py"
    click AICrawl "https://github.com/agent-on-the-fly/memento/blob/main/server/ai_crawl.py"
    click PageCrawl "https://github.com/agent-on-the-fly/memento/blob/main/server/craw_page.py"
    click DocTool "https://github.com/agent-on-the-fly/memento/blob/main/server/documents_tool.py"
    click ExcelTool "https://github.com/agent-on-the-fly/memento/blob/main/server/excel_tool.py"
    click MathTool "https://github.com/agent-on-the-fly/memento/blob/main/server/math_tool.py"
    click ImageTool "https://github.com/agent-on-the-fly/memento/blob/main/server/image_tool.py"
    click VideoTool "https://github.com/agent-on-the-fly/memento/blob/main/server/video_tool.py"
    click CodeAgent "https://github.com/agent-on-the-fly/memento/blob/main/server/code_agent.py"
    click BaseInterp "https://github.com/agent-on-the-fly/memento/blob/main/server/interpreters/base.py"
    click DockerInterp "https://github.com/agent-on-the-fly/memento/blob/main/server/interpreters/docker_interpreter.py"
    click SubpInterp "https://github.com/agent-on-the-fly/memento/blob/main/server/interpreters/subprocess_interpreter.py"
    click IPythonInterp "https://github.com/agent-on-the-fly/memento/blob/main/server/interpreters/ipython_interpreter.py"
    click InternalInterp "https://github.com/agent-on-the-fly/memento/blob/main/server/interpreters/internal_python_interpreter.py"
    click E2BInterp "https://github.com/agent-on-the-fly/memento/blob/main/server/interpreters/e2b_interpreter.py"
    click SearxNG "https://github.com/agent-on-the-fly/memento/tree/main/searxng-docker/"

    %% Styles
    classDef core fill:#D0E8FF,stroke:#0078D4,color:#000
    classDef tool fill:#DFFFD0,stroke:#228B22,color:#000
    classDef interp fill:#FFE4B5,stroke:#FF8C00,color:#000
    classDef external fill:#F0F0F0,stroke:#808080,color:#000
    classDef memory fill:#FFFACD,stroke:#DAA520,color:#000
```
