How themanojdesai/clicky works
1. Overall shape
Clicky is a macOS menu bar app backed by a small Cloudflare Worker proxy. The Swift app handles all local UX, permissions, audio/screen capture, and playback; the Worker keeps third-party API keys out of the app binary and exposes a few thin API endpoints.
2. Local app boundary
The main target is leanring-buddy in Xcode, built as a menu bar utility rather than a normal dock app. The UI is centered around NSPanel-style floating windows: a compact companion/control panel and a transparent overlay window used for cursor guidance.
3. Primary controller
CompanionManager.swift appears to be the central state/orchestration layer. It coordinates dictation, transcription, Claude requests, TTS playback, screen capture, overlay behavior, and app state transitions, making it the key architectural hub for diagramming.
4. Voice input pipeline
Push-to-talk and dictation are handled by BuddyDictationManager.swift, BuddyTranscriptionProvider.swift, and provider-specific implementations for Apple, AssemblyAI, and OpenAI. The app captures microphone audio locally, then either transcribes locally/on-device or streams audio to a remote transcription service depending on the selected path.
5. Remote transcription flow
For real-time transcription, the app uses AssemblyAI and OpenAI-specific adapters. The README indicates AssemblyAI token provisioning is proxied through the Worker, while audio streams over a websocket-based flow. This makes transcription one of the main networked subsystems in the app.
6. Screen understanding
The app can inspect the user’s current UI by capturing the screen and locating elements. CompanionScreenCaptureUtility.swift and ElementLocationDetector.swift suggest a capture-and-analysis pipeline that produces screenshots and coordinates for the model, enabling the “point at stuff” behavior.
7. LLM interaction
ClaudeAPI.swift is the chat/assistant client. The app sends the transcript plus screenshot/context to Claude using streaming SSE, then consumes streamed responses in real time. Claude is also allowed to emit point tags like [POINT:x,y:label:screenN], which drive the cursor overlay.
8. TTS and audio feedback
ElevenLabsTTSClient.swift handles voice playback for the assistant response. The README and file layout suggest the text response is converted to speech via ElevenLabs and played back locally, with supporting audio assets like enter.mp3, eshop.mp3, and ff.mp3 likely used for UI or interaction cues.
9. Cursor/overlay system
OverlayWindow.swift, CompanionResponseOverlay.swift, and WindowPositionManager.swift implement the visual guidance layer. This subsystem renders the movable blue cursor/overlay next to or over the user’s screen so the assistant can visually direct attention to specific UI locations.
10. Global interaction and permissions
GlobalPushToTalkShortcutMonitor.swift and the entitlement/plist files indicate the app relies on accessibility and screen-recording permissions for global hotkeys and screen access. This is an important architecture boundary: the app depends on macOS security capabilities to function as an always-available assistant.
11. Cloudflare Worker proxy
The worker/ package is a small TypeScript Cloudflare Worker with three proxy responsibilities: Claude chat, ElevenLabs TTS, and AssemblyAI token issuance. It is the network boundary between the macOS app and external AI services, and it holds the sensitive API keys via Worker secrets.
12. External services
The system integrates three main third-party services: Anthropic Claude for reasoning/chat, AssemblyAI for live transcription, and ElevenLabs for speech synthesis. The Worker mediates all of them, so the app itself only talks to one proxy host instead of directly to each vendor.
13. Supporting app infrastructure
AppBundleConfiguration.swift, ClickyAnalytics.swift, and DesignSystem.swift indicate supporting layers for app configuration, telemetry, and shared UI styling. These are cross-cutting concerns rather than primary product flows.
14. Repository composition
The repo is split into a macOS client, a Cloudflare Worker backend, and lightweight scripts. scripts/release.sh and appcast.xml suggest there is also a release/update distribution path, but the core runtime architecture is the client-plus-proxy model described above.
Main entry points: CompanionManager.swift Companion core · index.ts Worker proxy · leanring_buddyApp.swift App entry · BuddyTranscriptionProvider.swift Transcription API
Components 23
macOS app
Swift menu bar client and local interaction pipeline
- App entry · Menu bar app bootstrap and lifecycle
leanring-buddy/leanring_buddyApp.swift - Bundle config · App bundle and runtime configuration
leanring-buddy/AppBundleConfiguration.swift - Companion core · Central coordinator for dictation, screen capture, overlays, TTS, and state
leanring-buddy/CompanionManager.swift - Panel UI · Floating companion/control panel
leanring-buddy/CompanionPanelView.swift - Menu bar window · Menu bar panel/window presentation
leanring-buddy/MenuBarPanelManager.swift - Overlay system · Transparent cursor guidance and response overlay
leanring-buddy/OverlayWindow.swift - Response overlay · Renders assistant point/cursor guidance
leanring-buddy/CompanionResponseOverlay.swift - Window positioning · Positions floating windows and overlay elements
leanring-buddy/WindowPositionManager.swift - Dictation · Push-to-talk capture and transcription coordination
leanring-buddy/BuddyDictationManager.swift - Transcription API · Shared abstraction for speech-to-text providers
leanring-buddy/BuddyTranscriptionProvider.swift - Apple STT · On-device speech transcription path
leanring-buddy/AppleSpeechTranscriptionProvider.swift - AssemblyAI stream · Streaming transcription provider for live audio
leanring-buddy/AssemblyAIStreamingTranscriptionProvider.swift - OpenAI STT · OpenAI transcription adapter
leanring-buddy/OpenAIAudioTranscriptionProvider.swift - Screen capture · Captures screenshots for context and pointing
leanring-buddy/CompanionScreenCaptureUtility.swift - Element locator · Finds UI elements and coordinates in captured screens
leanring-buddy/ElementLocationDetector.swift - Claude client · Streams prompts and responses from Claude
leanring-buddy/ClaudeAPI.swift - TTS playback · ElevenLabs voice synthesis client
leanring-buddy/ElevenLabsTTSClient.swift - Audio support · Helpers for audio conversion and playback inputs
leanring-buddy/BuddyAudioConversionSupport.swift - Global hotkey · Push-to-talk shortcut monitor and global input hook
leanring-buddy/GlobalPushToTalkShortcutMonitor.swift - Analytics · Usage and event tracking
leanring-buddy/ClickyAnalytics.swift - Design system · Shared visual styling and UI constants
leanring-buddy/DesignSystem.swift
Cloudflare Worker
Thin proxy for third-party AI services
- Worker proxy · Cloudflare Worker entrypoint for vendor API mediation
worker/src/index.ts
Release tooling
Distribution and update helpers
- Release script · Build and publish helper for distribution
scripts/release.sh
Connections 29
- App entry → Companion core: boots · App lifecycle starts the main coordinator
- App entry → Menu bar window: creates · Bootstraps menu bar presentation
- App entry → Panel UI: shows · Displays companion UI
- App entry → Overlay system: opens · Initializes overlay windows
- Companion core → Dictation: controls · Starts and stops voice capture
- Companion core → Screen capture: captures · Requests screenshots and screen context
- Companion core → Claude client: queries · Sends transcript and context to the model
- Companion core → TTS playback: speaks · Converts assistant text to audio
- Companion core → Response overlay: guides · Feeds point tags and guidance to overlay rendering
- Companion core → Window positioning: positions · Coordinates floating UI placement
- Companion core → Analytics: tracks · Emits app usage telemetry
- Companion core → Design system: styles · Uses shared UI styling
- Dictation → Global hotkey: listens · Uses global push-to-talk shortcuts
- Dictation → Transcription API: selects · Routes audio to a transcription provider
- Transcription API → Apple STT: implements · Local transcription path
- Transcription API → AssemblyAI stream: implements · Streaming AssemblyAI path
- Transcription API → OpenAI STT: implements · OpenAI transcription path
- Screen capture → Element locator: analyzes · Produces UI element locations from screenshots
- Claude client → Worker proxy: proxies via · Uses the Worker for vendor API access
- TTS playback → Worker proxy: proxies via · Routes ElevenLabs requests through the Worker
- AssemblyAI stream → Worker proxy: token via · Obtains AssemblyAI token through the Worker
- Worker proxy → Claude client: forwards · Proxies Claude chat requests
- Worker proxy → TTS playback: forwards · Proxies ElevenLabs TTS requests
- Worker proxy → AssemblyAI stream: mints token · Issues AssemblyAI access token
- Overlay system → Response overlay: renders · Hosts the visual guidance overlay
- Menu bar window → Panel UI: presents · Displays the companion panel from menu-bar UI
- Window positioning → Overlay system: places · Computes overlay placement
- Audio support → TTS playback: supports · Provides audio conversion and playback support
- Release script → Worker proxy: deploys · Supports release/deployment flow