How receptron/mulmochat works
Project type and purpose
• MulmoChat is a full-stack web application: a research prototype for a “multimodal AI chat” interface.
• Frontend: Vue 3 + Vite SPA (canvas-based chat, interactive tools, voice chat).
• Backend: Node.js/TypeScript service exposing REST/WebSocket endpoints for text generation, tool invocation, and asset routes.
• Purpose: Orchestrate LLM calls and render rich visuals (maps, games, PDFs, image generation) inline with chat.
File structure
• Top-level dirs:
• server/ – Node TS server code, LLM abstraction, route handlers, tests
• src/ – Vue application (components, composables, tool plugins, views, previews)
• public/, index.html, Vite config, Tailwind config
• server/llm/providers – abstraction layer for Anthropic, OpenAI, Google Gemini, Ollama
• server/routes – express-style API endpoints: /api/text, /api/pdf, /api/movie, /api/comfyui, /api/html, /api/textLLM
• src/tools – plugin architecture: configs, logic, models, previews, views
• src/composables – abstractions for real-time session transport, text sessions, scrolling, user prefs
• Tests under server/tests to verify each provider and tool endpoint
README for additional insights
• Describes unified text API with GET /api/text/providers and POST /api/text/generate
• Explains how to wire API keys, run dev server, and sample test scripts
• References deeper architecture docs in WHITEPAPER.md and LLM_OS.md
Main components and interactions
a. Client UI Layer
• Vue SPA (App.vue, Sidebar.vue, GoogleMap.vue, canvas area)
• Composition API hooks (useTextSession, useRealtimeSession, useToolResults)
• Tool plugin system: dynamically renders tool UIs (tool previews → tool views)
• Static host: Vite dev server serves index.html + /public assets
b. API/Application Layer
• HTTP/WS endpoints in server/index.ts & server/routes/*
• TextService orchestrates provider calls, normalizes responses
• TextSessionStore holds conversation state (in-memory or persisted)
• ExaSearch wrapper for search, ComfyUI route for image workflows
c. LLM Provider Layer
• Abstract “provider” interface in server/llm/providers
• Concrete adapters for OpenAI, Anthropic, Google, Ollama
d. External Services
• APIs (Google Maps)
• EXA search API
• ComfyUI image pipelines
• PDF generation/processing
e. Data flows
1. User interacts via Vue UI → sends messages/events over REST or WebSocket
2. Server routes receive request → TextService or tool logic invoked
3. TextService calls provider adapter(s) → normalized response → TextSessionStore
4. Server responds to client → Vue updates canvas, tool render, or voice output
Architectural patterns and design principles
• Plugin architecture for tools: clear separation of config, logic, models, UI
• Adapter pattern for LLM providers: single interface, multiple implementations
• Composition API on frontend: encapsulate cross-cutting concerns (session transport, user prefs)
• Layered architecture: UI → API orchestration → provider integration → external services
• Test-driven verification: standalone scripts under server/tests for each provider and tool route