# rs-010806/evolve-mem architecture

> Architecture diagram of the GitHub repository rs-010806/evolve-mem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rs-010806/evolve-mem
- Repository: https://github.com/rs-010806/evolve-mem

## Overview

##### Project Type and Purpose
- EVOLVE-MEM is a research‐oriented, full‐stack Python system for self‐evolving hierarchical memory in agentic AI.
- It ingests raw experiences, embeds them, organizes them into a three‐tier memory hierarchy, serves queries at different abstraction levels, evaluates performance with SOTA metrics, and self‐improves over time.

##### File Structure
- Top‐level modules correspond to each major tier and function:
- dynamic_memory.py → Tier 1: embedding & storage
- hierarchical_manager.py → Tier 2: clustering & summarization
- self_improvement.py → Tier 3: performance monitoring & reorganization
- llm_backend.py → Gemini LLM integration
- memory_system.py → Main façade tying all tiers together
- evaluation.py & test_system_performance.py → Metrics & tests
- dataset_loader.py → Data ingestion
- run_comprehensive_pipeline.py → Orchestrates full research pipeline
- config.py, utils.py → Shared config & helper functions
- requirements.txt & .gitignore are standard. There are also directories for datasets, logs, and results.

##### Key Architectural Patterns
- Layered (three‐tier) memory hierarchy
- Pipeline/orchestration pattern for ingestion → organization → query → evaluation → improvement
- External service integration (Gemini LLM)
- Persistent storage (ChromaDB + JSON dumps)

##### Main Components
a. **Data Ingestion Layer**
- Dataset Loader (LoCoMo or custom)
- SentenceTransformer → Embedding
- In‐memory dictionary + ChromaDB (vector store)
b. **Tier 1: Dynamic Memory Network**
- Raw experience store (embeddings + metadata)
- Fast semantic search
c. **Tier 2: Hierarchical Memory Manager**
- Level 0 → raw notes
- Level 1 → KMeans clusters → LLM summaries
- Level 2 → Meta‐clusters → LLM abstractions
- JSON persistence for hierarchy
d. **Tier 3: Self‐Improvement Engine**
- Monitors accuracy, retrieval balance, latency
- Triggers re‐clustering, parameter tuning
e. **Query & Retrieval Engine**
- Query router (classifies by complexity)
- Multi‐level retrieval paths
- LLM answer extraction & post‐processing
- Logs retrieval path & similarity scores
f. **Evaluation & Metrics**
- Metric computations (Accuracy, BLEU-1, ROUGE, METEOR, SBERT)
- Reporting (results/, logs/)
g. **LLM Service**
- Gemini API (summarization, abstraction, reasoning)

##### Relationships and Data Flow
- Tier 1 → Tier 2 (clustering & summarization loops)
- Tier 2 → Tier 3 (self‐improvement feedback loop)
- External LLM used by Tier 2 & Query Engine
- Query Layer ↔ All memory tiers → returns answers
- Evaluation receives answers + ground truth → reports → feeds Self‐Improvement
- Pipeline orchestrator (run_comprehensive_pipeline.py) triggers all stages end‐to‐end

##### Architectural Patterns & Principles
- **Separation of Concerns:** distinct modules for memory storage, management, self‐improvement, LLM integration, evaluation
- **Feedback Loop:** evaluation → self‐improvement → memory reorganization
- **Persistent State:** vector DB + JSON dumps ensure reproducibility

- [ ] Highlight feedback loops (evaluation → self‐improvement → memory manager)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Storage Layer
    subgraph "Data Ingestion & Storage"
        direction TB
        DatasetLoader["Dataset Loader"]:::ingest
        Embedder["SentenceTransformer\n(Embedding)"]:::ingest
        InMemoryStore["In-memory Dictionary"]:::ingest
        ChromaDB[(ChromaDB)]:::ingest
    end

    %% Memory Processing Layers
    subgraph "Memory Processing" 
        direction TB
        Tier1["Dynamic Memory Network"]:::green
        Tier2["Hierarchical Memory Manager"]:::green
    end

    %% Self-Improvement & Evaluation
    subgraph "Self-Improvement & Evaluation"
        direction TB
        Tier3["Self-Improvement Engine"]:::orange
        Evaluation["Evaluation & Metrics"]:::orange
        Tests["Performance Tests"]:::orange
    end

    %% Query & Retrieval Engine
    QueryEngine["Query & Retrieval Engine"]:::green

    %% External Services
    subgraph "LLM & External Services"
        direction TB
        LLMService["Gemini LLM Service"]:::purple
    end

    %% Pipeline Orchestrator and Shared
    Pipeline["Pipeline Orchestrator"]:::orange
    Config["Shared Configuration"]:::infra
    Utils["Utility Helpers"]:::infra

    %% Data Flows
    Pipeline -->|orchestrates| DatasetLoader
    DatasetLoader -->|loads raw data| Embedder
    Embedder -->|creates embeddings| InMemoryStore
    InMemoryStore -->|stores vectors| ChromaDB
    ChromaDB -->|retrieves embeddings| Tier1
    Tier1 -->|raw notes| Tier2
    Tier2 -->|cluster summaries| Tier3
    Tier3 -->|optimization signals| Tier2
    Tier2 -->|abstract hierarchy| Tier3
    Tier2 -->|? uses LLM| LLMService
    QueryEngine -->|routes queries| Tier1
    QueryEngine -->|routes queries| Tier2
    QueryEngine -->|routes queries| Tier3
    QueryEngine -->|calls LLM| LLMService
    QueryEngine -->|returns answers| Evaluation
    Evaluation -->|feedback metrics| Tier3
    Tests -->|validates| Evaluation
    Pipeline --> Tier1
    Pipeline --> Tier2
    Pipeline --> Tier3
    Pipeline --> QueryEngine
    Pipeline --> Evaluation

    %% Shared Config and Utils usage
    Config --> DatasetLoader
    Config --> Tier1
    Config --> Tier2
    Config --> Tier3
    Config --> QueryEngine
    Utils --> DatasetLoader
    Utils --> Tier1
    Utils --> Tier2
    Utils --> Tier3
    Utils --> QueryEngine

    %% Click Events
    click DatasetLoader "https://github.com/rs-010806/evolve-mem/blob/main/dataset_loader.py"
    click Tier1 "https://github.com/rs-010806/evolve-mem/blob/main/dynamic_memory.py"
    click Tier2 "https://github.com/rs-010806/evolve-mem/blob/main/hierarchical_manager.py"
    click Tier3 "https://github.com/rs-010806/evolve-mem/blob/main/self_improvement.py"
    click QueryEngine "https://github.com/rs-010806/evolve-mem/blob/main/memory_system.py"
    click LLMService "https://github.com/rs-010806/evolve-mem/blob/main/llm_backend.py"
    click Evaluation "https://github.com/rs-010806/evolve-mem/blob/main/evaluation.py"
    click Tests "https://github.com/rs-010806/evolve-mem/blob/main/test_system_performance.py"
    click Pipeline "https://github.com/rs-010806/evolve-mem/blob/main/run_comprehensive_pipeline.py"
    click Config "https://github.com/rs-010806/evolve-mem/blob/main/config.py"
    click Utils "https://github.com/rs-010806/evolve-mem/blob/main/utils.py"

    %% Styles
    classDef ingest fill:#d0e7ff,stroke:#0366d6,color:#0366d6;
    classDef green fill:#d0ffd0,stroke:#228B22,color:#228B22;
    classDef orange fill:#ffe0b3,stroke:#ff8c00,color:#ff8c00;
    classDef purple fill:#e5d0ff,stroke:#800080,color:#800080;
    classDef infra fill:#f0f0f0,stroke:#666,color:#333;
```
