How vaggiri/ai-website-navigation-assistant works
This repository is a Chrome Extension + dual-backend AI service for “website navigation assistance”:
• In-page assistant (content script) renders a chat UI, captures text/voice queries, inspects the current page DOM/links, highlights the target link, and navigates.
• Dual-mode AI:
• Cloud mode: Node.js server calls Google Gemini for higher-level reasoning/general queries.
• Local mode: Python/Flask server hosts a SentenceTransformer embedding model for fast/offline semantic matching.
• Privacy/Firewall: extension blocks known trackers/third-party domains while allowing sensitive sites to work safely (rulesets / rules.json).
Chrome Extension (client-side, inside Chrome)
Key files:
• manifest.json — declares extension permissions, content scripts, background, host permissions, possibly rulesets.
• content.js — primary runtime logic injected into every webpage: UI, hotkeys (Shift+F, Esc), query handling, link scraping/highlighting/clicking, calling backends.
• styles.css — UI styling for the assistant overlay.
• background.js — extension background/service worker logic (often used for: network interception, rulesets, message routing, centralized state).
• popup.html, popup.js — extension popup UI (toggle modes, settings, mic, etc.).
• newtab.html — custom new-tab / onboarding or quick entry point.
Also present:
• debug_repro.js, test_gemini.js — dev/test utilities (don’t put in the main architecture view; optionally in a “Dev/Testing” subview).
Node.js “Cloud Mode” server (port 3000)
• server/server.js — HTTP API that receives user query/context and calls Gemini.
• server/package.json, package-lock.json — dependencies, runtime entry.
• .env (implied by README) — GEMINI_API_KEY.
Python “Local Mode” server (port 5000)
• server/ai_server.py — Flask API that loads a SentenceTransformer model and performs embedding similarity / link matching.
Local training pipeline (offline / dev workflow)
• server/training_data.json — supervised examples.
• server/train.py — fine-tuning script producing updated local model artifacts (not shown in tree, but implied).
User on arbitrary website
• content.js renders overlay UI (styled by styles.css).
User enters query (text or voice)
• Voice: Browser speech APIs or capture in content/popup (depending on implementation) → query text.
content.js extracts page context
• Collects visible links/buttons/nav items; may capture DOM text snippets.
content.js decides AI path
• Local mode → call http://localhost:5000/... (Flask)
• Cloud mode → call http://localhost:3000/... (Node)
Backend returns an answer
• Typically a best matching link label/URL/selector + confidence + explanation.
Throughout: Firewall/privacy shield blocks trackers
• Implemented via extension rules (ruleset/rules.json) and/or background interception.
This explicitly as:
• Content Script → (HTTP) → Node server → (HTTPS) → Gemini
• Content Script → (HTTP) → Python server → (local inference) → response
• Content Script → DOM highlight/click → Webpage
• Background / Rules Engine → block/allow network requests → affects Webpage loads and third-party calls
Cross-component messaging inside the extension
• popup.js ↔ background.js (mode toggle/settings stored centrally)
• popup.js ↔ content.js (open/close assistant, pass configuration)
• content.js ↔ background.js (optional: for permissions, storage, rules updates, centralized logging)
• Sidecar UX pattern: in-page overlay UI (content script) attached to arbitrary websites.
• Dual-backend strategy (policy-based routing):
• local fast semantic matching for privacy/offline
• cloud reasoning for general queries
• Edge privacy enforcement: request blocking at the extension layer using rulesets (rules.json, _metadata/...), likely via Chrome’s declarativeNetRequest model.
• Human-in-the-loop training: local model fine-tuning using training_data.json + train.py.
• User
• Chrome Extension
• Visited Website
• Node Cloud Server (3000)
• Python Local Server (5000)
• Google Gemini API
Plus “Tracker/3rd-party domains” as an external entity affected by the firewall.
• Chrome Extension container broken into:
• Content Script (content.js + styles.css)
• Background/Service Worker (background.js)
• Popup UI (popup.html/popup.js)
• Ruleset/Firewall (rules.json + _metadata)
• Node server container:
• REST endpoints
• Gemini client
• Env secrets
• Python server container:
• REST endpoints
• SentenceTransformer model
• Model artifacts
• Training pipeline as a separate container/lane (non-runtime)
content.js because it orchestrates:
• UI renderer
• Hotkey handler
• Voice capture → text
• Page link extraction (DOM scraper)
• Mode router (local vs cloud)
• Highlighter + navigator (scroll/click)
• Error handling/fallback heuristics (“change password” fast path)
• Browser / Extension
• User’s Webpage (DOM)
• Localhost Services
• External Cloud
This makes it instantly obvious what runs where and what’s remote.
• From content.js to backends:
“query + extracted_links/context + mode + (optional) site URL”
• From backends to content.js:
“target link/selector + confidence + rationale”
• Firewall:
“blocklist/allowlist rules” applied to “network requests”.
Ports and trust boundaries
Add:
• localhost:3000 Node
• localhost:5000 Python
• Trust boundary around “External: Gemini API” (keys, privacy).
• That “Cloud mode may transmit page context” (if it does)—mark as a privacy consideration.
Nodes at minimum:
• User
• Chrome (Extension)
• content.js (+ styles.css)
• background.js
• popup.html/popup.js
• rules.json + _metadata rulesets
• Visited Website (DOM)
• Node server (server/server.js, :3000) → Gemini API
• Python server (server/ai_server.py, :5000) → SentenceTransformer model
• Training lane: training_data.json → train.py → model artifacts → ai_server.py
• Third-party trackers (as an external entity affected by firewall)