How agent-on-the-fly/memento works
Project Type and Purpose
• This is a modular, agent-based system that implements a two-stage planner–executor architecture on top of large language models, augmented with a memory bank for continual learning. It is not a traditional full-stack web app but a distributed client/server framework:
• A lightweight “client” (the Planner–Executor agent)
• A set of “server” MCP (Multi-Context Protocol) tool servers
• A memory component for experience replay
• Integrations with external services (OpenAI LLMs, SearxNG, FFmpeg, SerpAPI, etc.)
File Structure Analysis
• Top-level directories:
• client/ – the agent entrypoint and CBR logic (Planner + Executor)
• server/ – individual MCP tool servers (code, search, document, image, video, etc.)
• interpreters/ – backends for sandboxed code execution (Docker, subprocess, IPython, internal Python)
• memory/ – memory implementations and case-bank data store
• data/ – sample input data and cases
• searxng-docker/ – configuration for SearxNG search engine
• Figure/, README.md, requirements, LICENSE, pyproject.toml
Key Components
a. Client Agent
• Meta-Planner module (task decomposition via GPT-4.1)
• Executor module (calls MCP tools)
• Case-Based Reasoning (CBR) layer (no-parametric memory)
• Memory adapter for Read/Write operations to Case Bank
b. MCP Tool Layer (server/)
• Web tools: search_tool, serp_search, ai_crawler, craw_page
• Data tools: documents_tool, excel_tool, math_tool
• Media tools: image_tool, video_tool
• Code tool: code_agent + interpreters
c. Interpreters
• BaseInterpreter + Docker, subprocess, IPython, internal Python, E2B
d. Memory
• Case Bank (memory/np_memory.py or dummy_memo.jsonl)
• Data store (data/deepresearcher.jsonl)
e. External Services
• OpenAI API (for planner & executor models)
• SearxNG search (via searxng-docker)
• FFmpeg (system binary for video)
• Optional APIs: Chunkr, Jina, AssemblyAI, SerpAPI
Relationships and Data Flows
• User Input → Client Agent (agent.py)
• Meta-Planner → OpenAI GPT endpoint → subtasks list
• Planner reads cases from Case Bank → refines subtask ordering
• Executor invokes MCP calls over HTTP/GRPC to server tools
• Server tools may spin up interpreters (for code), call SearxNG (search), FFmpeg (video), or external REST APIs
• Tool results → returned to Executor → aggregator → final response
• After each subtask or trajectory, the agent writes experience tuples to Case Bank
• Memory queries influence future Planner decisions
Architectural Patterns & Principles
• Decoupled Planner vs. Executor
• Memory-Augmented MDP modeled as CBR policy
• Microservice-style MCP servers for extensibility
• Interpreter abstraction for secure, sandboxed code execution
• Externalization of search engine via Docker (SearxNG)
Final Checklist
• All major components boxed and labeled
• Arrowheads indicating direction of calls and data flow
• Annotations for protocols (HTTP, MCP) and technologies
• A clear title: “Memento System Architecture”