How avarayr/yap-for-cursor works
Project type and purpose
• Type: A client-side editor extension (open-source tool) that injects JavaScript into the Cursor editor to provide in-browser, WebGPU-powered voice-to-text (ASR) using Hugging Face’s Whisper model.
• Purpose: Seamlessly add a microphone button and hotkey to Cursor’s UI, record audio locally, run a Whisper model in a Web Worker via WebGPU, and insert the transcribed text into the chat input.
File structure
• Top-level files
• build.ts / bun.lock / package.json / tsconfig.json: build and dependency configuration (Bundler = Bun, TypeScript).
• dist/yap-for-cursor.js: final injected bundle.
• src/
• src/main.ts: entry point that initializes the extension, loads UI, wires up hotkeys.
• src/ui/: UI components (mic-button.ts, context-menu.ts, dom-selectors.ts).
• src/asr/: ASR orchestration (manager.ts), instance lifecycle (instance.ts), Worker thread logic (worker.ts).
• src/audio/processing.ts: raw audio capture, preprocessing (e.g., buffering, format conversion).
• src/utils/hotkey.ts: global hotkey registration.
• src/config.ts & src/types.ts: configuration constants and shared type definitions.
• src/styles/: CSS for button styling.
• Build output in dist – ready-to-inject JS.
Key insights from README
• Local in-browser processing; no external servers except initial model download.
• Uses a VS Code “Custom CSS and JS Loader” to inject the bundle.
• Hotkey support (Cmd+Shift+Y) and click interactions.
• Dependencies: Hugging Face Transformers (Whisper), WebGPU, Web Worker API.
Main components & interactions (to represent
a. Cursor Host Environment
• The Cursor editor / DOM into which the script is injected via Custom CSS/JS Loader.
b. Injection Layer (dist/yap-for-cursor.js)
• The single bundle injected at startup.
c. UI Layer
• Microphone Button (src/ui/mic-button.ts)
• Context Menu (src/ui/context-menu.ts)
• DOM Selectors (src/ui/dom-selectors.ts)
• CSS Styles (src/styles/styles.css)
d. Control & Hotkey
• Hotkey Manager (src/utils/hotkey.ts)
• Entry Point (src/main.ts) wires UI↔Hotkey.
e. Audio Pipeline
• Audio Capture → Processing (src/audio/processing.ts) → PCM chunks
f. ASR Layer
• ASR Manager (src/asr/manager.ts): orchestrates instances & Worker
• ASR Instance (src/asr/instance.ts): per-recording state
• Web Worker (src/asr/worker.ts): runs Whisper via WebGPU
• WebGPU & Transformers Model
g. External Service
• Model Download from Hugging Face on first run
Relationships & data flow
1. UI Event (click hotkey) → main.ts
2. Audio Capture → audio/processing.ts → PCM stream
3. PCM → postMessage → ASR Worker
4. Worker loads Whisper via WebGPU → transcribes → returns text
5. Manager receives transcript → updates UI component
6. UI Layer injects text into Cursor chat input
Architectural patterns & principles
• Separation of Concerns: UI vs. Audio vs. ASR logic vs. config/types
• Asynchronous / Worker Offloading: Heavy ML runs off the main thread
• Modular TypeScript organized by feature (ui, asr, audio, utils)
• In-Browser Edge Computing: Local processing for privacy