How kessler/gemma-gem works
1. What this repository is
- This is a Chrome extension built with WXT that runs a browser-based AI assistant fully on-device.
- The main architectural split is between the page-facing UI/tools, the extension service worker, and an offscreen document that hosts the model and agent loop.
2. Primary runtime boundaries
- Content script: injected into web pages; owns the floating gem icon, the chat overlay, and page-local DOM interactions.
- Background service worker: acts as the message hub and handles browser-context operations that require extension privileges.
- Offscreen document: a hidden extension page used to load Gemma 4 via @huggingface/transformers and run inference with WebGPU.
3. Agent core
- The agent/ directory is the model-agnostic orchestration layer.
- It contains the loop that turns user requests plus page state into model prompts, parses tool calls from model output, and coordinates repeated tool/model iterations.
- The README notes this layer has zero dependencies and can be extracted as a standalone library, which makes it the cleanest subsystem boundary in the repo.
4. Tool execution split
- Tools are intentionally divided by execution context:
- Content script tools: read_page_content, click_element, type_text, scroll_page
- Background-only tools: take_screenshot, run_javascript
- This split matters architecturally because page DOM access is available in the content script, while privileged browser APIs and cross-context execution happen in the service worker.
5. Message routing
- The background/message-router.ts file is the central dispatcher between contexts.
- It connects content script requests to the offscreen model host and routes browser/tool requests in the opposite direction.
- Shared message contracts live in shared/messages.ts, which is the main interface boundary keeping the extension contexts in sync.
6. Model hosting
- The offscreen side (offscreen/model-host.ts and entrypoints/offscreen/) is responsible for loading and running the model locally.
- The model artifacts are stored in public/ort/, indicating ONNX Runtime Web assets for WebGPU/WASM execution.
- The model choice is user-configurable between Gemma 4 E2B and E4B, with the selected model persisted across sessions.
7. Content UI and page interaction
- The content layer (content/) injects a bottom-right gem icon and a shadow-DOM chat overlay.
- It is also the execution surface for page-affecting actions like reading content, clicking, typing, and scrolling.
- Because this code runs inside the page tab, it can directly observe and manipulate the DOM without going through the model host.
8. Background responsibilities
- The service worker handles extension-level orchestration rather than model work.
- Besides routing, it performs operations that need privileged access, such as screenshots and arbitrary JavaScript execution in the page context.
- background/offscreen-manager.ts indicates the worker also manages the lifecycle of the offscreen document.
9. Shared contract layer
- The shared/ directory holds the cross-cutting primitives:
- models.ts for model metadata/configuration
- tool-definitions.ts for the tool schema exposed to the agent
- messages.ts for inter-context messaging
- logger.ts for consistent log formatting
- This shared layer is what prevents the three runtime contexts from drifting apart.
10. Entry points and packaging
- entrypoints/background.ts, entrypoints/content.ts, and entrypoints/offscreen/ are the extension entry targets wired by WXT.
- wxt.config.ts and tsconfig.json define the build/runtime setup, while public/ contains assets shipped with the extension.
- The architecture is therefore a classic MV3 extension shape, but with a notable hidden-model runtime in the offscreen document.
11. End-to-end data flow
- User opens the overlay in a page tab.
- The content script collects page context and sends a request through the background router.
- The offscreen model host runs the agent loop, which may request tools.
- Tool calls are routed to either the content script or background worker depending on capability.
- Results flow back into the agent loop until a final assistant response is produced and rendered in the overlay.
12. Key architectural constraint
- The project is designed to keep all sensitive data local: page content, screenshots, and model inference stay inside the browser.
- That constraint shapes the whole repo: no external API service, no backend, and a strict separation between page DOM access, browser-privileged operations, and model execution.
Main entry points: agent-loop.ts Agent loop · message-router.ts Message router · model-host.ts Model host · chat-overlay.ts Chat overlay
Components 19
Runtime boundaries
Extension entrypoints and the three execution contexts: content script, background service worker, and offscreen model host.
- Background entry · Service worker bootstrap wired by WXT.
entrypoints/background.ts - Content entry · Injected page-side bootstrap for the overlay and DOM tools.
entrypoints/content.ts - Offscreen shell · Hidden extension document that hosts the model runtime.
entrypoints/offscreen/index.html - Offscreen main · Bootstraps the offscreen runtime.
entrypoints/offscreen/main.ts - Chat overlay · Shadow-DOM assistant panel shown inside the page.
content/chat-overlay.ts - Gem icon · Floating launcher injected into the page.
content/gem-icon.ts - Page tools · Page-local actions such as reading content, clicking, typing, and scrolling.
content/tool-executors.ts - Message router · Central dispatcher between content, offscreen, and privileged browser operations.
background/message-router.ts - Offscreen manager · Creates and maintains the offscreen document lifecycle.
background/offscreen-manager.ts - Model host · Loads Gemma locally and runs inference in the hidden document.
offscreen/model-host.ts
Agent core
Model-agnostic orchestration that builds prompts, parses tool calls, and iterates the loop.
- Agent loop · Coordinates prompt creation, model calls, tool execution, and repeated iterations.
agent/agent-loop.ts - Prompt builder · Assembles user request, page state, and tool context into model prompts.
agent/prompt-builder.ts - Tool parser · Extracts structured tool calls from model output.
agent/tool-parser.ts - Agent types · Core orchestration types used by the loop and parser.
agent/types.ts
Shared contracts
Cross-context message, model, tool, and logging definitions.
- Messages · Shared message contracts between extension contexts.
shared/messages.ts - Tool defs · Canonical tool definitions exposed to the agent.
shared/tool-definitions.ts - Models · Model metadata and persisted selection state.
shared/models.ts - Logger · Shared logging format for all extension contexts.
shared/logger.ts
Model assets
Locally shipped runtime assets for model execution.
- ORT assets · ONNX Runtime Web artifacts shipped with the extension.
public/ort
Connections 21
- Background entry → Message router: boots · Background entrypoint starts the service worker router.
- Background entry → Offscreen manager: initializes · Service worker wires the offscreen lifecycle manager.
- Content entry → Chat overlay: mounts · Content entry loads the assistant overlay.
- Content entry → Gem icon: injects · Content entry creates the floating launcher.
- Offscreen shell → Offscreen main: loads · Offscreen page loads its runtime script.
- Offscreen main → Model host: starts · Offscreen bootstrap initializes model hosting.
- Chat overlay → Page tools: uses · Overlay triggers page-local actions through the content executors.
- Gem icon → Chat overlay: opens · Launcher toggles the assistant panel.
- Message router → Messages: routes via · Dispatcher relies on shared message contracts.
- Message router → Offscreen manager: requests model · Router asks the manager to ensure the offscreen host exists.
- Message router → Page tools: forwards tools · Routes page-bound tool requests back to the content context.
- Message router → Model host: forwards prompts · Routes model-related messages to the offscreen host.
- Model host → Agent loop: runs · Model host executes the agent orchestration loop around inference.
- Model host → ORT assets: loads · Offscreen runtime uses local ORT Web assets.
- Agent loop → Prompt builder: builds · Loop asks the prompt builder to assemble context.
- Agent loop → Tool parser: parses · Loop interprets model output as tool calls.
- Agent loop → Tool defs: invokes · Loop is driven by the shared tool schema.
- Agent loop → Models: selects · Loop uses model metadata and persisted selection.
- Agent loop → Agent types: typed by · Core loop is constrained by shared agent types.
- Agent loop → Messages: communicates via · Agent loop exchanges results through shared message contracts.
- Messages → Logger: observed by · Shared logging is applied around cross-context messaging.