How anindyapal/desktop-cognition works
Step 1: Identify Project Type and Purpose
• This is a desktop application built with Electron (JavaScript/HTML/CSS) as the frontend, with a Python-based backend providing real-time transcription and benchmarking.
• Purpose: Capture system and microphone audio on macOS, chunk it, run Whisper-based transcription in real time (or benchmarks), and display results and “agents” (Meeting Agent, Sales Agent) in an invisible, persistent UI.
Step 2: Analyze File Structure
Top-level directories and files:
• macos-system-audio/ — native macOS audio capture (Swift, JS bridges)
• main.js, renderer.js, index.html, onboarding.html, styles.css — Electron main & renderer processes and UI
• electron_backend.py — entry-point for Python backend
• *.py scripts — tests (realtime_transcription_test.py, benchmark_test.py, quick_test.py)
• package.json & requirements.txt — JS and Python dependencies
• setup.sh — bootstrap script
Step 3: Extract Additional Insights from README
• Two main modes: real-time transcription (3-second chunks via Whisper variants) and benchmarking.
• Supports CPU/GPU, multiple model sizes, VAD filtering.
• macOS-specific audio-permission handling and native recorder integration.
A. Electron Application
1. Main Process (main.js)
• Launches BrowserWindow, manages lifecycle, spawns Python backend via child_process or python-shell.
• Handles app-level events (onboarding, permissions).
2. Renderer Process (renderer.js + index.html + styles.css)
• UI: displays transcripts, action items, sales insights.
• Sends user commands (start/stop, model selection) to Main via IPC.
B. Python Backend (electron_backend.py + whisper code)
• Subprocess started by Electron Main.
• Loads WhisperModel via faster_whisper (CTranslate2) or OpenAI Whisper.
• Implements: audio chunking, VAD filtering, model inference.
• Exposes JSON-over-stdin/stdout or WebSocket to Electron.
C. macOS System Audio Capture
• Native Swift recorder (macos-system-audio/Recorder.swift) wrapped by JS (permission.js, recording.js).
• Feeds raw audio either directly to Python or via Electron Main for forwarding.
D. Test & Benchmark Scripts (python)
• Standalone CLI for real-time tests and benchmarks; not part of UI code path but shares model-loading logic.
E. External Services & Dependencies
• Whisper models (local files)
• portaudio (native library)
• Node modules for Electron and IPC
Step 5: Relationships & Interactions
On App Start
UI (Renderer) → Main Process initializes → native permission prompt via macos-system-audio → user grants mic access.
Audio Capture
macOS Recorder → streams chunks to Python Backend.
Transcription Pipeline
Python Backend receives raw PCM chunks → apply VAD → invoke WhisperModel → returns text segments.
UI Update
Python Backend → send transcript JSON to Electron Main → forward via IPC to Renderer → display in UI.
Sales/Meeting Agent Logic
Renderer invokes Analysis logic (JS or Python) on streaming transcripts → highlights action items or missed cues.
Step 6: Architectural Patterns & Design Principles
• Process Isolation: Electron Main vs Renderer vs Python Backend
• IPC Channels: electron.ipcMain / ipcRenderer & child_process stdio
• Modular Native Extensions: Swift recorder code decoupled in its own directory
• Configurable Pipeline: Python code parameterizes model size, device, compute_type
For this full-stack desktop app:
• Components into “Electron Side” and “Python Side.”
• Native Recorder as an external module connected to both sides (permission in JS, data in Python).
– Renderer ⇄ Main (IPC)
– Main ⇄ Python Backend (child_process STDIO or sockets)
• Technology labels: Electron (Node.js, Chromium), Python 3.8+, faster-whisper, portaudio, Swift Recorder.
• Real-time flow: Audio Capture → Chunking → VAD → Inference → UI Render.
Step 8: Diagramming Guidelines
• Blue for Electron processes,
• Green for Python backend,
• Orange for Native/OS modules.
• Processes (rectangles),
• Native libraries (hexagons),
Step 9: Emphasize Detail
• Config flows (model selection, device flags) from UI to backend.
• Error/permission handling path: on permission denial, UI shows onboarding screen.