# aiming-lab/simplemem architecture

> Architecture diagram of the GitHub repository aiming-lab/simplemem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aiming-lab/simplemem
- Repository: https://github.com/aiming-lab/simplemem

## Overview

SimpleMem is a lifelong memory system for LLM agents: conversations or multimodal inputs become structured memories, are indexed, and are retrieved to answer later queries. The repository also includes MCP access, cross-session memory, and EvolveMem’s retrieval-optimization capabilities. The README documents the high-level workflow and product branches; source-index references support several internal dependencies. Many subsystems are unsampled, so their detailed call paths are omitted rather than inferred.

Main entry points: [`simplemem/multimodal/app.py`](https://github.com/aiming-lab/simplemem/blob/main/simplemem/multimodal/app.py) (Multimodal API), [`simplemem/router.py`](https://github.com/aiming-lab/simplemem/blob/main/simplemem/router.py) (SimpleMem API), [`simplemem/core/memory_builder.py`](https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/memory_builder.py) (Memory builder), [`simplemem/core/hybrid_retriever.py`](https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/hybrid_retriever.py) (Hybrid retrieval)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Text memory"]
  node_simplemem["SimpleMem API<br/>[router.py]"]
  node_builder["Memory builder<br/>[memory_builder.py]"]
  node_vector[("Vector store<br/>[vector_store.py]")]
  node_retriever["Hybrid retrieval"]
  node_answer["Answer generation"]
  node_llm["LLM client<br/>[llm_client.py]"]
end

subgraph group_multi["Multimodal memory"]
  node_multiapp["Multimodal API<br/>[app.py]"]
  node_processors["Media processors"]
  node_multimodal_store[("Multimodal storage")]
  node_knowledge["Knowledge graph"]
  node_retrieval_multi["Multimodal retrieval"]
end

subgraph group_access["Agent access"]
  node_mcp["MCP server<br/>[mcp_handler.py]"]
  node_http["HTTP server<br/>[http_server.py]"]
end

subgraph group_cross["Cross-session"]
  node_cross_orch["Session orchestrator<br/>[orchestrator.py]"]
  node_cross_store[("Session storage<br/>[storage_sqlite.py]")]
end

subgraph group_evolve["Memory evolution"]
  node_evolver["Evolution manager<br/>[manager.py]"]
  node_evolve_store[("Evolution store<br/>[store.py]")]
  node_experience["Meta-experience"]
end

node_agent(("LLM agent"))
node_mcp_client(("MCP client"))

node_agent -->|"submits memory"| node_simplemem
node_simplemem -.->|"stores dialogue"| node_builder
node_builder -.->|"indexes memories"| node_vector
node_simplemem -.->|"requests recall"| node_retriever
node_retriever -->|"searches memories"| node_vector
node_simplemem -->|"generates answer"| node_answer
node_answer -->|"calls model"| node_llm
node_agent -->|"sends media"| node_multiapp
node_multiapp -.->|"processes modalities"| node_processors
node_multiapp -.->|"stores media memory"| node_multimodal_store
node_multiapp -.->|"builds knowledge graph"| node_knowledge
node_multiapp -.->|"retrieves memories"| node_retrieval_multi
node_mcp_client -->|"invokes tools"| node_mcp
node_mcp -.->|"serves requests"| node_http
node_cross_orch -.->|"persists sessions"| node_cross_store
node_evolver -->|"reads and writes"| node_evolve_store
node_experience -.->|"records outcomes"| node_experience

click node_simplemem "https://github.com/aiming-lab/simplemem/blob/main/simplemem/router.py"
click node_builder "https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/memory_builder.py"
click node_vector "https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/database/vector_store.py"
click node_retriever "https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/hybrid_retriever.py"
click node_answer "https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/answer_generator.py"
click node_llm "https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/utils/llm_client.py"
click node_multiapp "https://github.com/aiming-lab/simplemem/blob/main/simplemem/multimodal/app.py"
click node_processors "https://github.com/aiming-lab/simplemem/tree/main/simplemem/multimodal/processors"
click node_multimodal_store "https://github.com/aiming-lab/simplemem/tree/main/simplemem/multimodal/storage"
click node_knowledge "https://github.com/aiming-lab/simplemem/tree/main/simplemem/multimodal/knowledge"
click node_retrieval_multi "https://github.com/aiming-lab/simplemem/tree/main/simplemem/multimodal/retrieval"
click node_mcp "https://github.com/aiming-lab/simplemem/blob/main/MCP/server/mcp_handler.py"
click node_http "https://github.com/aiming-lab/simplemem/blob/main/MCP/server/http_server.py"
click node_cross_orch "https://github.com/aiming-lab/simplemem/blob/main/cross/orchestrator.py"
click node_cross_store "https://github.com/aiming-lab/simplemem/blob/main/cross/storage_sqlite.py"
click node_evolver "https://github.com/aiming-lab/simplemem/blob/main/simplemem/evolver/manager.py"
click node_evolve_store "https://github.com/aiming-lab/simplemem/blob/main/simplemem/evolver/store.py"
click node_experience "https://github.com/aiming-lab/simplemem/blob/main/simplemem/multimodal/evolution/experience_engine.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_simplemem,node_builder,node_vector,node_retriever,node_answer,node_llm,node_mcp_client toneBlue
class node_multiapp,node_processors,node_multimodal_store,node_knowledge,node_retrieval_multi toneAmber
class node_mcp,node_http toneMint
class node_cross_orch,node_cross_store toneRose
class node_evolver,node_evolve_store,node_experience,node_agent toneIndigo
```

## Components

### Text memory

- [SimpleMem API](https://github.com/aiming-lab/simplemem/blob/main/simplemem/router.py): `simplemem/router.py`
- [Memory builder](https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/memory_builder.py): `simplemem/core/memory_builder.py`
- [Vector store](https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/database/vector_store.py): `simplemem/core/database/vector_store.py`
- [Hybrid retrieval](https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/hybrid_retriever.py): `simplemem/core/hybrid_retriever.py`
- [Answer generation](https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/answer_generator.py): `simplemem/core/answer_generator.py`
- [LLM client](https://github.com/aiming-lab/simplemem/blob/main/simplemem/core/utils/llm_client.py): `simplemem/core/utils/llm_client.py`

### Multimodal memory

- [Multimodal API](https://github.com/aiming-lab/simplemem/blob/main/simplemem/multimodal/app.py): `simplemem/multimodal/app.py`
- [Media processors](https://github.com/aiming-lab/simplemem/tree/main/simplemem/multimodal/processors): `simplemem/multimodal/processors`
- [Multimodal storage](https://github.com/aiming-lab/simplemem/tree/main/simplemem/multimodal/storage): `simplemem/multimodal/storage`
- [Knowledge graph](https://github.com/aiming-lab/simplemem/tree/main/simplemem/multimodal/knowledge): `simplemem/multimodal/knowledge`
- [Multimodal retrieval](https://github.com/aiming-lab/simplemem/tree/main/simplemem/multimodal/retrieval): `simplemem/multimodal/retrieval`

### Agent access

- [MCP server](https://github.com/aiming-lab/simplemem/blob/main/MCP/server/mcp_handler.py): `MCP/server/mcp_handler.py`
- [HTTP server](https://github.com/aiming-lab/simplemem/blob/main/MCP/server/http_server.py): `MCP/server/http_server.py`

### Cross-session

- [Session orchestrator](https://github.com/aiming-lab/simplemem/blob/main/cross/orchestrator.py): `cross/orchestrator.py`
- [Session storage](https://github.com/aiming-lab/simplemem/blob/main/cross/storage_sqlite.py): `cross/storage_sqlite.py`

### Memory evolution

- [Evolution manager](https://github.com/aiming-lab/simplemem/blob/main/simplemem/evolver/manager.py): `simplemem/evolver/manager.py`
- [Evolution store](https://github.com/aiming-lab/simplemem/blob/main/simplemem/evolver/store.py): `simplemem/evolver/store.py`
- [Meta-experience](https://github.com/aiming-lab/simplemem/blob/main/simplemem/multimodal/evolution/experience_engine.py): `simplemem/multimodal/evolution/experience_engine.py`

### Other components

- **LLM agent**
- **MCP client**

## Connections

- LLM agent → SimpleMem API: submits memory (evidence: `README.md`)
- SimpleMem API → Memory builder: stores dialogue
- Memory builder → Vector store: indexes memories (evidence: `simplemem/core/memory_builder.py`)
- SimpleMem API → Hybrid retrieval: requests recall
- Hybrid retrieval → Vector store: searches memories (evidence: `simplemem/core/hybrid_retriever.py`)
- SimpleMem API → Answer generation: generates answer
- Answer generation → LLM client: calls model (evidence: `simplemem/core/answer_generator.py`)
- LLM agent → Multimodal API: sends media (evidence: `README.md`)
- Multimodal API → Media processors: processes modalities
- Multimodal API → Multimodal storage: stores media memory
- Multimodal API → Knowledge graph: builds knowledge graph
- Multimodal API → Multimodal retrieval: retrieves memories
- MCP client → MCP server: invokes tools (evidence: `README.md`)
- MCP server → HTTP server: serves requests
- Session orchestrator → Session storage: persists sessions
- Evolution manager → Evolution store: reads and writes (evidence: `simplemem/evolver/manager.py`)
- Meta-experience → Meta-experience: records outcomes (evidence: `simplemem/multimodal/evolution/experience_engine.py`)
