# receptron/mulmochat architecture

> Architecture diagram of the GitHub repository receptron/mulmochat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/receptron/mulmochat
- Repository: https://github.com/receptron/mulmochat

## Overview

##### Project type and purpose
- MulmoChat is a full-stack web application: a research prototype for a “multimodal AI chat” interface.
- Frontend: Vue 3 + Vite SPA (canvas-based chat, interactive tools, voice chat).
- Backend: Node.js/TypeScript service exposing REST/WebSocket endpoints for text generation, tool invocation, and asset routes.
- Purpose: Orchestrate LLM calls and render rich visuals (maps, games, PDFs, image generation) inline with chat.

##### File structure
- Top-level dirs:
- server/ – Node TS server code, LLM abstraction, route handlers, tests
- src/ – Vue application (components, composables, tool plugins, views, previews)
- public/, index.html, Vite config, Tailwind config
- server/llm/providers – abstraction layer for Anthropic, OpenAI, Google Gemini, Ollama
- server/routes – express-style API endpoints: /api/text, /api/pdf, /api/movie, /api/comfyui, /api/html, /api/textLLM
- src/tools – plugin architecture: configs, logic, models, previews, views
- src/composables – abstractions for real-time session transport, text sessions, scrolling, user prefs
- Tests under server/tests to verify each provider and tool endpoint

##### README for additional insights
- Describes unified text API with GET /api/text/providers and POST /api/text/generate
- Explains how to wire API keys, run dev server, and sample test scripts
- References deeper architecture docs in WHITEPAPER.md and LLM_OS.md

##### Main components and interactions
a. Client UI Layer
- Vue SPA (App.vue, Sidebar.vue, GoogleMap.vue, canvas area)
- Composition API hooks (useTextSession, useRealtimeSession, useToolResults)
- Tool plugin system: dynamically renders tool UIs (tool previews → tool views)
- Static host: Vite dev server serves index.html + /public assets
b. API/Application Layer
- HTTP/WS endpoints in server/index.ts & server/routes/*
- TextService orchestrates provider calls, normalizes responses
- TextSessionStore holds conversation state (in-memory or persisted)
- ExaSearch wrapper for search, ComfyUI route for image workflows
c. LLM Provider Layer
- Abstract “provider” interface in server/llm/providers
- Concrete adapters for OpenAI, Anthropic, Google, Ollama
d. External Services
- APIs (Google Maps)
- EXA search API
- ComfyUI image pipelines
- PDF generation/processing
e. Data flows
1. User interacts via Vue UI → sends messages/events over REST or WebSocket
2. Server routes receive request → TextService or tool logic invoked
3. TextService calls provider adapter(s) → normalized response → TextSessionStore
4. Server responds to client → Vue updates canvas, tool render, or voice output

##### Architectural patterns and design principles
- Plugin architecture for tools: clear separation of config, logic, models, UI
- Adapter pattern for LLM providers: single interface, multiple implementations
- Composition API on frontend: encapsulate cross-cutting concerns (session transport, user prefs)
- Layered architecture: UI → API orchestration → provider integration → external services
- Test-driven verification: standalone scripts under server/tests for each provider and tool route

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Client Layer
  subgraph "Browser / Client"
    direction TB
    AppVue["App.vue"]:::client
    Sidebar["Sidebar.vue"]:::client
    GoogleMapComp["GoogleMap.vue"]:::client
    TextSelect["TextSelectionMenu.vue"]:::client
    MainEntry["main.ts"]:::client
    Styles["index.css"]:::client
    ViteConfig["vite.config.ts"]:::client
    TailwindConfig["tailwind.config.cjs"]:::client

    subgraph "Composition API Hooks"
      direction TB
      useTextSession["useTextSession.ts"]:::client
      useRealtimeSession["useRealtimeSession.ts"]:::client
      useToolResults["useToolResults.ts"]:::client
      useSessionTransport["useSessionTransport.ts"]:::client
      useUserPreferences["useUserPreferences.ts"]:::client
      useVoiceRealtimeSession["useVoiceRealtimeSession.ts"]:::client
      useScrolling["useScrolling.ts"]:::client
    end

    subgraph "Tool Plugin System"
      direction TB
      ToolRegistry["index.ts (Registry)"]:::client
      ToolConfig["ImageGenerationConfig.vue"]:::client
      LogicGo["goLogic.ts"]:::client
      LogicOthello["othelloLogic.ts"]:::client
      ToolModels["models/*.ts"]:::client
      ToolPreviews["previews/*.vue"]:::client
      ToolViews["views/*.vue"]:::client
      ToolTypes["types.ts"]:::client
    end
  end

  %% API / Application Layer
  subgraph "API Layer"
    direction TB
    ServerEntry["server/index.ts"]:::server
    subgraph "Routes"
      direction TB
      TextLLM["textLLM.ts (Unified Text)"]:::server
      PDFRoute["pdf.ts"]:::server
      MovieRoute["movie.ts"]:::server
      HTMLRoute["html.ts"]:::server
      ComfyUIRoute["comfyui.ts"]:::server
    end
    TextService["TextService.ts"]:::server
    SessionStore["TextSessionStore"]:::server
    EXAWrapper["exaSearch.ts"]:::server
  end

  %% Provider Layer
  subgraph "LLM Provider Layer"
    direction TB
    Anthropic["anthropic.ts"]:::provider
    OpenAIAdapter["openai.ts"]:::provider
    GoogleAdapter["google.ts"]:::provider
    Ollama["ollama.ts"]:::provider
    ProviderTypes["types.ts"]:::provider
  end

  %% External Services
  subgraph "External Services"
    direction TB
    ExtOpenAI["OpenAI API"]:::external
    ExtAnthropic["Anthropic API"]:::external
    ExtGoogleMaps["Google Maps"]:::external
    ExtEXA["EXA Search API"]:::external
    ExtComfyUI["ComfyUI"]:::external
  end

  %% Data Flows
  AppVue -->|"REST/WebSocket events"| TextLLM
  TextSelect -->|"POST /api/text/generate"| TextLLM
  GoogleMapComp -->|"GET /api/html"| HTMLRoute
  ToolRegistry -->|"invoke tool endpoint"| MovieRoute
  TextLLM -->|"calls"| TextService
  TextService -->|"uses"| SessionStore
  TextService -->|"adapter call"| Anthropic
  TextService -->|"adapter call"| OpenAIAdapter
  TextService -->|"adapter call"| GoogleAdapter
  TextService -->|"adapter call"| Ollama
  Anthropic -->|"API request"| ExtAnthropic
  OpenAIAdapter -->|"API request"| ExtOpenAI
  GoogleAdapter -->|"API request"| ExtGoogleMaps
  Ollama -->|"API request"| ExtComfyUI
  ComfyUIRoute -->|"image pipeline"| ExtComfyUI
  EXAWrapper -->|"search"| ExtEXA
  SessionStore -->|"state updates"| AppVue

  %% Click Events for Client
  click AppVue "https://github.com/receptron/mulmochat/blob/main/src/App.vue"
  click Sidebar "https://github.com/receptron/mulmochat/blob/main/src/components/Sidebar.vue"
  click GoogleMapComp "https://github.com/receptron/mulmochat/blob/main/src/components/GoogleMap.vue"
  click TextSelect "https://github.com/receptron/mulmochat/blob/main/src/components/TextSelectionMenu.vue"
  click MainEntry "https://github.com/receptron/mulmochat/blob/main/src/main.ts"
  click Styles "https://github.com/receptron/mulmochat/blob/main/src/index.css"
  click ViteConfig "https://github.com/receptron/mulmochat/blob/main/vite.config.ts"
  click TailwindConfig "https://github.com/receptron/mulmochat/blob/main/tailwind.config.cjs"

  click useTextSession "https://github.com/receptron/mulmochat/blob/main/src/composables/useTextSession.ts"
  click useRealtimeSession "https://github.com/receptron/mulmochat/blob/main/src/composables/useRealtimeSession.ts"
  click useToolResults "https://github.com/receptron/mulmochat/blob/main/src/composables/useToolResults.ts"
  click useSessionTransport "https://github.com/receptron/mulmochat/blob/main/src/composables/useSessionTransport.ts"
  click useUserPreferences "https://github.com/receptron/mulmochat/blob/main/src/composables/useUserPreferences.ts"
  click useVoiceRealtimeSession "https://github.com/receptron/mulmochat/blob/main/src/composables/useVoiceRealtimeSession.ts"
  click useScrolling "https://github.com/receptron/mulmochat/blob/main/src/composables/useScrolling.ts"

  click ToolRegistry "https://github.com/receptron/mulmochat/blob/main/src/tools/index.ts"
  click ToolConfig "https://github.com/receptron/mulmochat/blob/main/src/tools/configs/ImageGenerationConfig.vue"
  click LogicGo "https://github.com/receptron/mulmochat/blob/main/src/tools/logic/goLogic.ts"
  click LogicOthello "https://github.com/receptron/mulmochat/blob/main/src/tools/logic/othelloLogic.ts"
  click ToolModels "https://github.com/receptron/mulmochat/tree/main/src/tools/models"
  click ToolPreviews "https://github.com/receptron/mulmochat/tree/main/src/tools/previews"
  click ToolViews "https://github.com/receptron/mulmochat/tree/main/src/tools/views"
  click ToolTypes "https://github.com/receptron/mulmochat/blob/main/src/tools/types.ts"

  %% Click Events for Server
  click ServerEntry "https://github.com/receptron/mulmochat/blob/main/server/index.ts"
  click TextLLM "https://github.com/receptron/mulmochat/blob/main/server/routes/textLLM.ts"
  click PDFRoute "https://github.com/receptron/mulmochat/blob/main/server/routes/pdf.ts"
  click MovieRoute "https://github.com/receptron/mulmochat/blob/main/server/routes/movie.ts"
  click HTMLRoute "https://github.com/receptron/mulmochat/blob/main/server/routes/html.ts"
  click ComfyUIRoute "https://github.com/receptron/mulmochat/blob/main/server/routes/comfyui.ts"
  click TextService "https://github.com/receptron/mulmochat/blob/main/server/llm/textService.ts"
  click SessionStore "https://github.com/receptron/mulmochat/blob/main/server/llm/textSessionStore.ts"
  click EXAWrapper "https://github.com/receptron/mulmochat/blob/main/server/exaSearch.ts"
  click Anthropic "https://github.com/receptron/mulmochat/blob/main/server/llm/providers/anthropic.ts"
  click OpenAIAdapter "https://github.com/receptron/mulmochat/blob/main/server/llm/providers/openai.ts"
  click GoogleAdapter "https://github.com/receptron/mulmochat/blob/main/server/llm/providers/google.ts"
  click Ollama "https://github.com/receptron/mulmochat/blob/main/server/llm/providers/ollama.ts"
  click ProviderTypes "https://github.com/receptron/mulmochat/blob/main/server/llm/types.ts"

  %% Styles
  classDef client fill:#D0E6FF,stroke:#0366d6,stroke-width:1px
  classDef server fill:#D8F5D0,stroke:#228B22,stroke-width:1px
  classDef provider fill:#FFE3B8,stroke:#FF8C00,stroke-width:1px
  classDef external fill:#E0E0E0,stroke:#888888,stroke-width:1px
```
