# kessler/gemma-gem architecture

> Architecture diagram of the GitHub repository kessler/gemma-gem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kessler/gemma-gem
- Repository: https://github.com/kessler/gemma-gem

## Overview

1. **What this repository is**
   - This is a Chrome extension built with **WXT** that runs a browser-based AI assistant fully **on-device**.
   - The main architectural split is between the **page-facing UI/tools**, the **extension service worker**, and an **offscreen document** that hosts the model and agent loop.

2. **Primary runtime boundaries**
   - **Content script**: injected into web pages; owns the floating gem icon, the chat overlay, and page-local DOM interactions.
   - **Background service worker**: acts as the message hub and handles browser-context operations that require extension privileges.
   - **Offscreen document**: a hidden extension page used to load **Gemma 4** via `@huggingface/transformers` and run inference with **WebGPU**.

3. **Agent core**
   - The `agent/` directory is the model-agnostic orchestration layer.
   - It contains the loop that turns user requests plus page state into model prompts, parses tool calls from model output, and coordinates repeated tool/model iterations.
   - The README notes this layer has zero dependencies and can be extracted as a standalone library, which makes it the cleanest subsystem boundary in the repo.

4. **Tool execution split**
   - Tools are intentionally divided by execution context:
     - **Content script tools**: `read_page_content`, `click_element`, `type_text`, `scroll_page`
     - **Background-only tools**: `take_screenshot`, `run_javascript`
   - This split matters architecturally because page DOM access is available in the content script, while privileged browser APIs and cross-context execution happen in the service worker.

5. **Message routing**
   - The `background/message-router.ts` file is the central dispatcher between contexts.
   - It connects content script requests to the offscreen model host and routes browser/tool requests in the opposite direction.
   - Shared message contracts live in `shared/messages.ts`, which is the main interface boundary keeping the extension contexts in sync.

6. **Model hosting**
   - The offscreen side (`offscreen/model-host.ts` and `entrypoints/offscreen/`) is responsible for loading and running the model locally.
   - The model artifacts are stored in `public/ort/`, indicating **ONNX Runtime Web** assets for WebGPU/WASM execution.
   - The model choice is user-configurable between **Gemma 4 E2B** and **E4B**, with the selected model persisted across sessions.

7. **Content UI and page interaction**
   - The content layer (`content/`) injects a bottom-right gem icon and a shadow-DOM chat overlay.
   - It is also the execution surface for page-affecting actions like reading content, clicking, typing, and scrolling.
   - Because this code runs inside the page tab, it can directly observe and manipulate the DOM without going through the model host.

8. **Background responsibilities**
   - The service worker handles extension-level orchestration rather than model work.
   - Besides routing, it performs operations that need privileged access, such as screenshots and arbitrary JavaScript execution in the page context.
   - `background/offscreen-manager.ts` indicates the worker also manages the lifecycle of the offscreen document.

9. **Shared contract layer**
   - The `shared/` directory holds the cross-cutting primitives:
     - `models.ts` for model metadata/configuration
     - `tool-definitions.ts` for the tool schema exposed to the agent
     - `messages.ts` for inter-context messaging
     - `logger.ts` for consistent log formatting
   - This shared layer is what prevents the three runtime contexts from drifting apart.

10. **Entry points and packaging**
   - `entrypoints/background.ts`, `entrypoints/content.ts`, and `entrypoints/offscreen/` are the extension entry targets wired by WXT.
   - `wxt.config.ts` and `tsconfig.json` define the build/runtime setup, while `public/` contains assets shipped with the extension.
   - The architecture is therefore a classic MV3 extension shape, but with a notable hidden-model runtime in the offscreen document.

11. **End-to-end data flow**
   - User opens the overlay in a page tab.
   - The content script collects page context and sends a request through the background router.
   - The offscreen model host runs the agent loop, which may request tools.
   - Tool calls are routed to either the content script or background worker depending on capability.
   - Results flow back into the agent loop until a final assistant response is produced and rendered in the overlay.

12. **Key architectural constraint**
   - The project is designed to keep all sensitive data local: page content, screenshots, and model inference stay inside the browser.
   - That constraint shapes the whole repo: no external API service, no backend, and a strict separation between page DOM access, browser-privileged operations, and model execution.

Main entry points: [`agent/agent-loop.ts`](https://github.com/kessler/gemma-gem/blob/main/agent/agent-loop.ts) (Agent loop), [`background/message-router.ts`](https://github.com/kessler/gemma-gem/blob/main/background/message-router.ts) (Message router), [`offscreen/model-host.ts`](https://github.com/kessler/gemma-gem/blob/main/offscreen/model-host.ts) (Model host), [`content/chat-overlay.ts`](https://github.com/kessler/gemma-gem/blob/main/content/chat-overlay.ts) (Chat overlay)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime boundaries"]
  node_entry_bg["Background entry<br/>wxt entrypoint<br/>[background.ts]"]
  node_entry_content["Content entry<br/>wxt entrypoint<br/>[content.ts]"]
  node_entry_offscreen_html["Offscreen shell<br/>offscreen page<br/>[index.html]"]
  node_entry_offscreen_main["Offscreen main<br/>offscreen entrypoint<br/>[main.ts]"]
  node_content_overlay["Chat overlay<br/>content UI<br/>[chat-overlay.ts]"]
  node_content_icon(("Gem icon<br/>content UI<br/>[gem-icon.ts]"))
  node_content_tools["Page tools<br/>content executors<br/>[tool-executors.ts]"]
  node_bg_router["Message router<br/>service worker router<br/>[message-router.ts]"]
  node_bg_offscreen["Offscreen manager<br/>lifecycle manager"]
  node_model_host[("Model host<br/>offscreen runtime<br/>[model-host.ts]")]
end

subgraph group_agent["Agent core"]
  node_agent_loop["Agent loop<br/>orchestrator<br/>[agent-loop.ts]"]
  node_prompt_builder["Prompt builder<br/>prompting<br/>[prompt-builder.ts]"]
  node_tool_parser["Tool parser<br/>parsing<br/>[tool-parser.ts]"]
  node_agent_types["Agent types<br/>contracts<br/>[types.ts]"]
end

subgraph group_shared["Shared contracts"]
  node_shared_msgs["Messages<br/>cross-context schema<br/>[messages.ts]"]
  node_shared_tools["Tool defs<br/>tool schema"]
  node_shared_models["Models<br/>model config<br/>[models.ts]"]
  node_shared_logger["Logger<br/>logging<br/>[logger.ts]"]
end

subgraph group_assets["Model assets"]
  node_ort_assets[("ORT assets<br/>runtime assets")]
end

node_entry_bg -->|"boots"| node_bg_router
node_entry_bg -->|"initializes"| node_bg_offscreen
node_entry_content -->|"mounts"| node_content_overlay
node_entry_content -->|"injects"| node_content_icon
node_entry_offscreen_html -->|"loads"| node_entry_offscreen_main
node_entry_offscreen_main -->|"starts"| node_model_host
node_content_overlay -->|"uses"| node_content_tools
node_content_icon -->|"opens"| node_content_overlay
node_bg_router -->|"routes via"| node_shared_msgs
node_bg_router -->|"requests model"| node_bg_offscreen
node_bg_router -->|"forwards tools"| node_content_tools
node_bg_router -->|"forwards prompts"| node_model_host
node_model_host -->|"runs"| node_agent_loop
node_model_host -->|"loads"| node_ort_assets
node_agent_loop -->|"builds"| node_prompt_builder
node_agent_loop -->|"parses"| node_tool_parser
node_agent_loop -->|"invokes"| node_shared_tools
node_agent_loop -->|"selects"| node_shared_models
node_agent_loop -->|"typed by"| node_agent_types
node_agent_loop -->|"communicates via"| node_shared_msgs
node_shared_msgs -.->|"observed by"| node_shared_logger

click node_entry_bg "https://github.com/kessler/gemma-gem/blob/main/entrypoints/background.ts"
click node_entry_content "https://github.com/kessler/gemma-gem/blob/main/entrypoints/content.ts"
click node_entry_offscreen_html "https://github.com/kessler/gemma-gem/blob/main/entrypoints/offscreen/index.html"
click node_entry_offscreen_main "https://github.com/kessler/gemma-gem/blob/main/entrypoints/offscreen/main.ts"
click node_content_overlay "https://github.com/kessler/gemma-gem/blob/main/content/chat-overlay.ts"
click node_content_icon "https://github.com/kessler/gemma-gem/blob/main/content/gem-icon.ts"
click node_content_tools "https://github.com/kessler/gemma-gem/blob/main/content/tool-executors.ts"
click node_bg_router "https://github.com/kessler/gemma-gem/blob/main/background/message-router.ts"
click node_bg_offscreen "https://github.com/kessler/gemma-gem/blob/main/background/offscreen-manager.ts"
click node_model_host "https://github.com/kessler/gemma-gem/blob/main/offscreen/model-host.ts"
click node_agent_loop "https://github.com/kessler/gemma-gem/blob/main/agent/agent-loop.ts"
click node_prompt_builder "https://github.com/kessler/gemma-gem/blob/main/agent/prompt-builder.ts"
click node_tool_parser "https://github.com/kessler/gemma-gem/blob/main/agent/tool-parser.ts"
click node_agent_types "https://github.com/kessler/gemma-gem/blob/main/agent/types.ts"
click node_shared_msgs "https://github.com/kessler/gemma-gem/blob/main/shared/messages.ts"
click node_shared_tools "https://github.com/kessler/gemma-gem/blob/main/shared/tool-definitions.ts"
click node_shared_models "https://github.com/kessler/gemma-gem/blob/main/shared/models.ts"
click node_shared_logger "https://github.com/kessler/gemma-gem/blob/main/shared/logger.ts"
click node_ort_assets "https://github.com/kessler/gemma-gem/tree/main/public/ort"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_entry_bg,node_entry_content,node_entry_offscreen_html,node_entry_offscreen_main,node_content_overlay,node_content_icon,node_content_tools,node_bg_router,node_bg_offscreen,node_model_host toneBlue
class node_agent_loop,node_prompt_builder,node_tool_parser,node_agent_types toneAmber
class node_shared_msgs,node_shared_tools,node_shared_models,node_shared_logger toneMint
class node_ort_assets toneRose
```

## Components

### Runtime boundaries

Extension entrypoints and the three execution contexts: content script, background service worker, and offscreen model host.

- [Background entry](https://github.com/kessler/gemma-gem/blob/main/entrypoints/background.ts): wxt entrypoint · Service worker bootstrap wired by WXT. · `entrypoints/background.ts`
- [Content entry](https://github.com/kessler/gemma-gem/blob/main/entrypoints/content.ts): wxt entrypoint · Injected page-side bootstrap for the overlay and DOM tools. · `entrypoints/content.ts`
- [Offscreen shell](https://github.com/kessler/gemma-gem/blob/main/entrypoints/offscreen/index.html): offscreen page · Hidden extension document that hosts the model runtime. · `entrypoints/offscreen/index.html`
- [Offscreen main](https://github.com/kessler/gemma-gem/blob/main/entrypoints/offscreen/main.ts): offscreen entrypoint · Bootstraps the offscreen runtime. · `entrypoints/offscreen/main.ts`
- [Chat overlay](https://github.com/kessler/gemma-gem/blob/main/content/chat-overlay.ts): content UI · Shadow-DOM assistant panel shown inside the page. · `content/chat-overlay.ts`
- [Gem icon](https://github.com/kessler/gemma-gem/blob/main/content/gem-icon.ts): content UI · Floating launcher injected into the page. · `content/gem-icon.ts`
- [Page tools](https://github.com/kessler/gemma-gem/blob/main/content/tool-executors.ts): content executors · Page-local actions such as reading content, clicking, typing, and scrolling. · `content/tool-executors.ts`
- [Message router](https://github.com/kessler/gemma-gem/blob/main/background/message-router.ts): service worker router · Central dispatcher between content, offscreen, and privileged browser operations. · `background/message-router.ts`
- [Offscreen manager](https://github.com/kessler/gemma-gem/blob/main/background/offscreen-manager.ts): lifecycle manager · Creates and maintains the offscreen document lifecycle. · `background/offscreen-manager.ts`
- [Model host](https://github.com/kessler/gemma-gem/blob/main/offscreen/model-host.ts): offscreen runtime · Loads Gemma locally and runs inference in the hidden document. · `offscreen/model-host.ts`

### Agent core

Model-agnostic orchestration that builds prompts, parses tool calls, and iterates the loop.

- [Agent loop](https://github.com/kessler/gemma-gem/blob/main/agent/agent-loop.ts): orchestrator · Coordinates prompt creation, model calls, tool execution, and repeated iterations. · `agent/agent-loop.ts`
- [Prompt builder](https://github.com/kessler/gemma-gem/blob/main/agent/prompt-builder.ts): prompting · Assembles user request, page state, and tool context into model prompts. · `agent/prompt-builder.ts`
- [Tool parser](https://github.com/kessler/gemma-gem/blob/main/agent/tool-parser.ts): parsing · Extracts structured tool calls from model output. · `agent/tool-parser.ts`
- [Agent types](https://github.com/kessler/gemma-gem/blob/main/agent/types.ts): contracts · Core orchestration types used by the loop and parser. · `agent/types.ts`

### Shared contracts

Cross-context message, model, tool, and logging definitions.

- [Messages](https://github.com/kessler/gemma-gem/blob/main/shared/messages.ts): cross-context schema · Shared message contracts between extension contexts. · `shared/messages.ts`
- [Tool defs](https://github.com/kessler/gemma-gem/blob/main/shared/tool-definitions.ts): tool schema · Canonical tool definitions exposed to the agent. · `shared/tool-definitions.ts`
- [Models](https://github.com/kessler/gemma-gem/blob/main/shared/models.ts): model config · Model metadata and persisted selection state. · `shared/models.ts`
- [Logger](https://github.com/kessler/gemma-gem/blob/main/shared/logger.ts): logging · Shared logging format for all extension contexts. · `shared/logger.ts`

### Model assets

Locally shipped runtime assets for model execution.

- [ORT assets](https://github.com/kessler/gemma-gem/tree/main/public/ort): runtime assets · ONNX Runtime Web artifacts shipped with the extension. · `public/ort`

## Connections

- Background entry → Message router: boots · Background entrypoint starts the service worker router.
- Background entry → Offscreen manager: initializes · Service worker wires the offscreen lifecycle manager.
- Content entry → Chat overlay: mounts · Content entry loads the assistant overlay.
- Content entry → Gem icon: injects · Content entry creates the floating launcher.
- Offscreen shell → Offscreen main: loads · Offscreen page loads its runtime script.
- Offscreen main → Model host: starts · Offscreen bootstrap initializes model hosting.
- Chat overlay → Page tools: uses · Overlay triggers page-local actions through the content executors.
- Gem icon → Chat overlay: opens · Launcher toggles the assistant panel.
- Message router → Messages: routes via · Dispatcher relies on shared message contracts.
- Message router → Offscreen manager: requests model · Router asks the manager to ensure the offscreen host exists.
- Message router → Page tools: forwards tools · Routes page-bound tool requests back to the content context.
- Message router → Model host: forwards prompts · Routes model-related messages to the offscreen host.
- Model host → Agent loop: runs · Model host executes the agent orchestration loop around inference.
- Model host → ORT assets: loads · Offscreen runtime uses local ORT Web assets.
- Agent loop → Prompt builder: builds · Loop asks the prompt builder to assemble context.
- Agent loop → Tool parser: parses · Loop interprets model output as tool calls.
- Agent loop → Tool defs: invokes · Loop is driven by the shared tool schema.
- Agent loop → Models: selects · Loop uses model metadata and persisted selection.
- Agent loop → Agent types: typed by · Core loop is constrained by shared agent types.
- Agent loop → Messages: communicates via · Agent loop exchanges results through shared message contracts.
- Messages → Logger: observed by · Shared logging is applied around cross-context messaging.
