# emojumprepo/worldseek-memory architecture

> Architecture diagram of the GitHub repository emojumprepo/worldseek-memory, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/emojumprepo/worldseek-memory
- Repository: https://github.com/emojumprepo/worldseek-memory

## Overview

##### Project type and purpose
- This is a full-stack AI Agent Platform (“FastGPT”) in a monorepo, combining:
- A Next.js-based frontend (projects/app) for UI & SSR/CSR
- A Node.js API layer (pages/api) backed by a “service” module
- Background workers for CPU/IO-heavy tasks (service/worker, BullMQ)
- Microservices (projects/mcp_server, projects/sandbox)
- Plugin & template runtimes (packages/plugins, packages/templates)
- A documentation site (docSite)
- Helm/Docker deployment manifests

##### File structure
- Top-level directories denote layers:
- packages/global, packages/service, packages/web: shared libraries
- packages/plugins, packages/templates: dynamic extensions
- projects/app: the main Next.js UI
- projects/mcp_server & sandbox: standalone microservices
- service/worker: background job processors
- service/common, core, support: layered backend business logic
- deploy/docker & deploy/helm: Kubernetes/Docker deployment

##### README for insights
- Core features: workflow orchestration, knowledge base storage (MongoDB + vector DB via PGVector/Milvus), OpenAPI interface, embeddable chat, plugin & template marketplace
- Tech stack: Next.js, TypeScript, ChakraUI, MongoDB, PostgreSQL (with PGVector) or Milvus, Redis, BullMQ, Helm/Kubernetes

##### Main components & their interactions
a. Frontend (Next.js UI)
- Renders pages & components (projects/app/src/pages & packages/web)
- Calls API endpoints under /api (hosted in Next.js)
- Embeds chat widget via an Iframe/API
b. API Server (Next.js API + service module)
- HTTP endpoints implement core operations (chat, dataset, app, workflow)
- Uses service/common for middleware (CORS, auth, rate-limit)
- Delegates to service/core for business logic (AI calls, dataset management, workflow dispatch)
- Persists data in MongoDB (sessions, logs, configs) and Postgres/Milvus (vector indices)
- Uses Redis for caching & BullMQ as a job queue
c. Background Workers
- service/worker actors consume BullMQ queues
- Perform tasks like embedding generation, file parsing (PDF→text), transcription, vector indexing
- Write results back to DB or signal completion
d. Microservices
- mcp_server: lightweight proxy for MCP tools, external integrations
- sandbox: code‐sandbox service for secure JS execution in workflows
e. Plugin & Template Runtime
- packages/plugins: dynamically loadable user plugins run in a worker/VM
- packages/templates: prebuilt prompt templates shown in UI
f. External Services
- AI providers via one-api or custom aiproxy
- Third-party data sources (webcrawler, OpenAPI integrations, outLink)
- Authentication/SSO via external identity providers
g. Data Stores
- MongoDB: primary document store (user data, chat logs, configs)
- PostgreSQL + PGVector or Milvus: vector databases for semantic search
- Redis: cache & BullMQ backend

##### Architectural patterns & design principles
- Layered architecture: common → core → support → third-provider modules
- Monorepo with pnpm workspaces for code sharing
- Microservices for isolation of sandbox and MCP proxy
- Serverless‐style API in Next.js for rapid iteration
- Event-driven background jobs via BullMQ queues
- Plugin architecture: user‐provided code runs in isolated runtime
- Kubernetes/Helm for deployment and scaling

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% UI Layer
    subgraph "UI Layer"
        direction TB
        UI["Next.js UI<br>(projects/app)"]:::frontend
        WebUI["Shared Web UI<br>(packages/web)"]:::share
        UI --> WebUI
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        API["Next.js API<br>(projects/app/src/pages/api)"]:::api
        CommonUtil["Global Utilities<br>(packages/global)"]:::share
        ServiceCommon["Service Common<br>(packages/service/common)"]:::service
        ServiceCore["Service Core<br>(packages/service/core)"]:::service
        ServiceSupport["Service Support<br>(packages/service/support)"]:::service

        API --> CommonUtil
        API --> ServiceCommon
        ServiceCommon --> ServiceCore
        ServiceCore --> ServiceSupport
    end

    %% Worker Layer
    subgraph "Worker Layer"
        direction TB
        Workers["BullMQ Workers<br>(packages/service/worker)"]:::worker
    end

    %% Microservices
    subgraph "Microservices"
        direction TB
        MCP["MCP Proxy<br>(projects/mcp_server)"]:::api
        Sandbox["Sandbox Service<br>(projects/sandbox)"]:::api
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        Mongo["MongoDB"]:::datastore
        Postgres["PostgreSQL<br>+ PGVector/Milvus"]:::datastore
        Redis["Redis<br>(Cache & Queue)"]:::datastore
        BullMQ["BullMQ Queue"]:::datastore
    end

    %% External Services
    subgraph "External Services"
        direction TB
        AI["AI Providers"]:::ext
        SSO["SSO/Identity"]:::ext
        ThirdParty["Third-Party Data"]:::ext
    end

    %% Documentation & Deployment
    subgraph "Docs & Deployment"
        direction TB
        DocSite["Documentation Site<br>(docSite)"]:::share
        Docker["Docker Manifests<br>(deploy/docker)"]:::share
        Helm["Helm Charts<br>(deploy/helm/fastgpt)"]:::share
    end

    %% Flow Connections
    UI -->|"calls API"| API
    API -->|"enqueue job"| BullMQ
    API -->|"read/write"| Mongo
    API -->|"read/write"| Postgres
    API -->|"cache"| Redis
    API -->|"auth via"| SSO
    API -->|"invoke external"| ThirdParty

    Workers -->|"dequeue job"| BullMQ
    Workers -->|"read/write"| Mongo
    Workers -->|"read/write"| Postgres
    Workers -->|"cache"| Redis
    Workers -->|"call AI"| AI

    MCP -->|"proxy requests"| ThirdParty
    Sandbox -->|"execute code"| ThirdParty

    %% Shared packages usage
    CommonUtil -.-> ServiceCommon
    CommonUtil -.-> API
    ServiceCore -.-> Workers

    %% User interaction
    User["User"]:::frontend
    User --> UI

    %% Click Events
    click UI "https://github.com/emojumprepo/worldseek-memory/tree/main/projects/app"
    click WebUI "https://github.com/emojumprepo/worldseek-memory/tree/main/packages/web"
    click API "https://github.com/emojumprepo/worldseek-memory/tree/main/projects/app/src/pages/api"
    click CommonUtil "https://github.com/emojumprepo/worldseek-memory/tree/main/packages/global"
    click ServiceCommon "https://github.com/emojumprepo/worldseek-memory/tree/main/packages/service/common"
    click ServiceCore "https://github.com/emojumprepo/worldseek-memory/tree/main/packages/service/core"
    click ServiceSupport "https://github.com/emojumprepo/worldseek-memory/tree/main/packages/service/support"
    click Workers "https://github.com/emojumprepo/worldseek-memory/tree/main/packages/service/worker"
    click MCP "https://github.com/emojumprepo/worldseek-memory/tree/main/projects/mcp_server"
    click Sandbox "https://github.com/emojumprepo/worldseek-memory/tree/main/projects/sandbox"
    click DocSite "https://github.com/emojumprepo/worldseek-memory/tree/main/docSite"
    click Docker "https://github.com/emojumprepo/worldseek-memory/tree/main/deploy/docker"
    click Helm "https://github.com/emojumprepo/worldseek-memory/tree/main/deploy/helm/fastgpt"

    %% Styles
    classDef frontend fill:#cfe2ff,stroke:#0366d6,color:#000
    classDef api fill:#d1e7dd,stroke:#0f5132,color:#000
    classDef worker fill:#ffe5b4,stroke:#d38303,color:#000
    classDef datastore fill:#d3d3d3,stroke:#6c757d,color:#000
    classDef ext fill:#e4d0ff,stroke:#6f42c1,color:#000
    classDef service fill:#e2f0d9,stroke:#198754,color:#000
    classDef share fill:#f0f0f0,stroke:#adb5bd,color:#000
```
