How andrewnguonly/lumos works
Project Type & Purpose
• This is a Chrome extension that acts as a RAG (Retrieval Augmented Generation) LLM co-pilot for web browsing.
• It relies on a local LLM (powered via Ollama) to provide inference and embedding capabilities.
• The extension integrates chat features, content parsing (with customizable configs), multimodal file processing, and tool invocation (like a calculator).
File Structure
• Top-level files & folders include configuration files (.eslintrc, manifest.json, tsconfig.json, webpack.config.js), documentation (docs/), tests (__tests__/), assets (screenshots, src/assets), and source code (src/).
• The src folder houses:
• The entry point (index.tsx) and primary React component (App.tsx), with various subdirectories for components (ChatBar, ChatHistory, CodeBlock), contexts (ThemeContext), pages (Popup, Options), document loaders, tools, and local vector stores.
• Scripts such as background.ts and content.ts that likely serve as the background script and content script for the Chrome extension.
• The presence of a manifest.json confirms it is a Chrome extension, while options.tsx and popup.tsx define its UI pages.
Key Architectural Components
• Chrome Extension Components:
• Manifest: Acts as the entry to define the extension’s permissions and entry points.
• Background Script (src/scripts/background.ts): Runs as a long-lived process handling events and managing communications.
• Content Script (src/scripts/content.ts): Interacts with the current webpage, possibly for content parsing and multimodal support.
• UI Components: Popup (src/pages/Popup.tsx) and Options (src/pages/Options.tsx) pages along with various React components (ChatBar, ChatHistory, CodeBlock).
• Contexts (src/contexts/ThemeContext.tsx): Providing application-wide shared state (e.g., light/dark theme).
• Local LLM Integration:
• The extension communicates with a locally running Ollama server (as detailed in the README). This server handles:
• LLM inference.
• Embedding model inference.
• Possibly vectorstore operations (src/vectorstores/enhanced_memory.ts).
• RAG Pipeline:
• Document Loaders (src/document_loaders/): Modules like csv.ts and dynamic_file.ts prepare input text (or files such as CSV) for the LLM.
• Content Parser & Config (src/contentConfig.ts): Customizable per-URL behavior (see docs/content_parser.md) that feeds data into the RAG vectorstore.
• Tools (src/tools/calculator.ts): Additional helpers that process specific prompt types.
• External Integrations & Workflows:
• The detailed README explains support for file attachments, multimodal processing (image handling in docs/multimodal.md), and third-party guidance (e.g., LangChain).
Relationships and Interactions
• UI Components (Popup, Options) → communicate user actions (prompt, configuration changes).
• Data/process flows from the content script and background script → to external calls made to the local Ollama server.
• Responses (LLM-generated answers, embeddings, and tool responses) flow back into the extension’s UI.
• The vector store (enhanced_memory.ts) is used as part of the retrieval pipeline when handling long documents or summarizations.
• The document loaders feed parsed content into the RAG process.
• Asynchronous, event-driven design common to Chrome extensions, where the background and content scripts manage state and relay messages between the UI and the local server.