How dcrebbin/meta-vision-api works
Project type and purpose
• This is a lightweight full-stack “hacky” integration:
• A browser-side bookmarklet (client) that watches Facebook Messenger for new photo messages and forwards their URLs
• A small REST API server (backend) built in TypeScript (running on Bun) that relays image URLs to the OpenAI GPT-4 Vision API, stores the results, and exposes them as static JSON
File structure
• Top-level files:
• package.json, bun.lockb, tsconfig.json, .env.example → Node/Bun project config
• src/index.ts → server entry point (defines REST endpoint /api/gpt-4-vision)
• src/bookmarklet.js → client-side JS to be injected into messenger.com via a bookmark
• public/ → static serving directory (contains data.json after the server writes results)
• .gitignore, README.md → meta/config
README for insights
• Explains prerequisites (Meta Ray-Ban Smart Glasses + Messenger account + OpenAI key)
• Describes workflow: voice commands → Messenger chat on alternative account → bookmarklet observes new images → POST to local server → GPT-4 Vision call → write to public/data.json → read and display on glasses
Components and interactions
a. Main components
1. User & Meta Ray-Ban Smart Glasses (voice input / display)
2. Meta View App (on glasses)
3. Browser (messenger.com) with injected Bookmarklet code
4. Bookmarklet Observer (in-page JS)
5. Local REST API Server (Bun + TypeScript)
6. OpenAI GPT-4 Vision API (external)
7. Static Data Store (public/data.json)
b. Relationships & data flow
• User “says” voice command → Meta View App determines recipient
• Meta View App sends Messenger photo message to alt account
• In browser (messenger.com), BookmarkletObserver listens for new <img> nodes
• On new image: Observer ⟶ HTTP POST /api/gpt-4-vision (JSON with imageUrl)
• Server receives POST ⟶ forwards image URL to OpenAI GPT-4 Vision API
• On success: server writes the JSON result to public/data.json
• Glasses / Meta View App periodically fetches or is notified to fetch data.json to render the analysis on the display
c. Architectural patterns & principles
• Client-side injection (bookmarklet) as a lightweight “micro-frontend”
• Simple REST API façade pattern: server proxies to external AI service
• Static file serving for results → no database required
d. Key technologies
• Bookmarklet (vanilla JS + DOMNodeInserted event)
• Bun runtime, TypeScript, Express-style routing (in index.ts)
• OpenAI GPT-4 Vision API
• Static file hosting (via Bun’s static middleware or equivalent)
Elements
• Voice command → Meta View App → Messenger → Browser → Bookmarklet → Server → OpenAI → Server → JSON store → Glasses
• External services (OpenAI GPT-4 Vision, Meta services) → grayed out
Detailed capture of architectural elements
• User & Glasses vs. Meta View App vs. Messenger web UI vs. Bookmarklet observer
• REST API vs. OpenAI API vs. file store
• Use notes or annotations for limitations (DOMNodeInserted hack, CORS mode=no-cors, environment variables)