# anindyapal/desktop-cognition architecture

> Architecture diagram of the GitHub repository anindyapal/desktop-cognition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anindyapal/desktop-cognition
- Repository: https://github.com/anindyapal/desktop-cognition

## Overview

Step 1: Identify Project Type and Purpose
- This is a desktop application built with Electron (JavaScript/HTML/CSS) as the frontend, with a Python-based backend providing real-time transcription and benchmarking.
- Purpose: Capture system and microphone audio on macOS, chunk it, run Whisper-based transcription in real time (or benchmarks), and display results and “agents” (Meeting Agent, Sales Agent) in an invisible, persistent UI.

Step 2: Analyze File Structure
Top-level directories and files:
- macos-system-audio/ — native macOS audio capture (Swift, JS bridges)
- main.js, renderer.js, index.html, onboarding.html, styles.css — Electron main & renderer processes and UI
- electron_backend.py — entry-point for Python backend
- *.py scripts — tests (realtime_transcription_test.py, benchmark_test.py, quick_test.py)
- package.json & requirements.txt — JS and Python dependencies
- setup.sh — bootstrap script

Step 3: Extract Additional Insights from README
- Two main modes: real-time transcription (3-second chunks via Whisper variants) and benchmarking.
- Supports CPU/GPU, multiple model sizes, VAD filtering.
- macOS-specific audio-permission handling and native recorder integration.

A. Electron Application
1. Main Process (main.js)
- Launches BrowserWindow, manages lifecycle, spawns Python backend via child_process or python-shell.
- Handles app-level events (onboarding, permissions).
2. Renderer Process (renderer.js + index.html + styles.css)
- UI: displays transcripts, action items, sales insights.
- Sends user commands (start/stop, model selection) to Main via IPC.

B. Python Backend (electron_backend.py + whisper code)
- Subprocess started by Electron Main.
- Loads WhisperModel via faster_whisper (CTranslate2) or OpenAI Whisper.
- Implements: audio chunking, VAD filtering, model inference.
- Exposes JSON-over-stdin/stdout or WebSocket to Electron.

C. macOS System Audio Capture
- Native Swift recorder (macos-system-audio/Recorder.swift) wrapped by JS (permission.js, recording.js).
- Feeds raw audio either directly to Python or via Electron Main for forwarding.

D. Test & Benchmark Scripts (python)
- Standalone CLI for real-time tests and benchmarks; not part of UI code path but shares model-loading logic.

E. External Services & Dependencies
- Whisper models (local files)
- portaudio (native library)
- Node modules for Electron and IPC

Step 5: Relationships & Interactions
##### On App Start
UI (Renderer) → Main Process initializes → native permission prompt via macos-system-audio → user grants mic access.
##### Audio Capture
macOS Recorder → streams chunks to Python Backend.
##### Transcription Pipeline
Python Backend receives raw PCM chunks → apply VAD → invoke WhisperModel → returns text segments.
##### UI Update
Python Backend → send transcript JSON to Electron Main → forward via IPC to Renderer → display in UI.
##### Sales/Meeting Agent Logic
Renderer invokes Analysis logic (JS or Python) on streaming transcripts → highlights action items or missed cues.

Step 6: Architectural Patterns & Design Principles
- Process Isolation: Electron Main vs Renderer vs Python Backend
- IPC Channels: electron.ipcMain / ipcRenderer & child_process stdio
- Modular Native Extensions: Swift recorder code decoupled in its own directory
- Configurable Pipeline: Python code parameterizes model size, device, compute_type

For this full-stack desktop app:
- Components into “Electron Side” and “Python Side.”
- Native Recorder as an external module connected to both sides (permission in JS, data in Python).
– Renderer ⇄ Main (IPC)
– Main ⇄ Python Backend (child_process STDIO or sockets)
- Technology labels: Electron (Node.js, Chromium), Python 3.8+, faster-whisper, portaudio, Swift Recorder.
- Real-time flow: Audio Capture → Chunking → VAD → Inference → UI Render.

Step 8: Diagramming Guidelines
- Blue for Electron processes,
- Green for Python backend,
- Orange for Native/OS modules.
- Processes (rectangles),
- Native libraries (hexagons),

Step 9: Emphasize Detail
- Config flows (model selection, device flags) from UI to backend.
- Error/permission handling path: on permission denial, UI shows onboarding screen.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Electron Side
    subgraph "Electron Side"
        direction TB
        EM["Main Process"]:::electron
        ER["Renderer Process"]:::electron
        UI1["index.html"]:::electron
        UI2["onboarding.html"]:::electron
        STY["styles.css"]:::electron
    end

    %% Python Side
    subgraph "Python Side"
        direction TB
        PB["Transcription & Benchmarking Service"]:::python
        RT["Real-Time Mode"]:::python
        BM["Benchmark Mode"]:::python
    end

    %% Native macOS Audio Capture
    subgraph "Native macOS Audio Capture"
        direction TB
        SW["Recorder.swift"]:::native
        PJ["permission.js"]:::native
        RJ["recording.js"]:::native
        RR["RealtimeRecorder"]:::native
        RC["Recorder"]:::native
    end

    %% Test Scripts
    subgraph "Test & Scripts"
        direction TB
        T1["realtime_transcription_test.py"]:::python
        T2["benchmark_test.py"]:::python
        T3["quick_test.py"]:::python
        BS["setup.sh"]:::native
        PJ2["package.json & package-lock.json"]:::electron
        REQ["requirements.txt"]:::python
        PT["prompt_general.txt\nprompt_sales.txt\nprompt_summary.txt"]:::native
        AS["assets/icon.icns"]:::native
    end

    %% External Dependencies
    WH[("Whisper Models")]:::storage
    PA[[portaudio]]:::external
    NM[[Node Modules]]:::external

    %% Connections
    ER -->|loads UI| UI1
    ER -->|loads UI| UI2
    ER -->|loads CSS| STY

    ER <-->|IPC Commands| EM
    EM -->|spawn child_process| PB
    EM -->|request permission| PJ
    PJ -->|invoke native prompt| SW
    SW -->|permission granted| EM
    SW -->|permission denied| ER
    EM -->|forward audio chunks| PB
    RJ -->|stream audio| PB
    RR -->|native module| PB
    RC -->|support| PB

    PB -->|returns transcript| EM
    EM -->|send transcript| ER
    ER -->|display transcript| UI1

    ER -->|agent analysis prompts| PB
    RT -->|uses Whisper & VAD| PB
    BM -->|runs benchmarks| PB

    PB -->|access models| WH
    PB -->|use portaudio| PA
    EM -->|use Node modules| NM

    %% Test Scripts Connections
    T1 -->|test real-time| RT
    T2 -->|test benchmarking| BM
    T3 -->|sanity test| PB

    %% Click Events
    click EM "https://github.com/anindyapal/desktop-cognition/blob/main/main.js"
    click ER "https://github.com/anindyapal/desktop-cognition/blob/main/renderer.js"
    click UI1 "https://github.com/anindyapal/desktop-cognition/blob/main/index.html"
    click UI2 "https://github.com/anindyapal/desktop-cognition/blob/main/onboarding.html"
    click STY "https://github.com/anindyapal/desktop-cognition/blob/main/styles.css"
    click PB "https://github.com/anindyapal/desktop-cognition/blob/main/electron_backend.py"
    click SW "https://github.com/anindyapal/desktop-cognition/blob/main/macos-system-audio/Recorder.swift"
    click PJ "https://github.com/anindyapal/desktop-cognition/blob/main/macos-system-audio/permission.js"
    click RJ "https://github.com/anindyapal/desktop-cognition/blob/main/macos-system-audio/recording.js"
    click RR "https://github.com/anindyapal/desktop-cognition/tree/main/macos-system-audio/RealtimeRecorder/"
    click RC "https://github.com/anindyapal/desktop-cognition/tree/main/macos-system-audio/Recorder/"
    click T1 "https://github.com/anindyapal/desktop-cognition/blob/main/realtime_transcription_test.py"
    click T2 "https://github.com/anindyapal/desktop-cognition/blob/main/benchmark_test.py"
    click T3 "https://github.com/anindyapal/desktop-cognition/blob/main/quick_test.py"
    click BS "https://github.com/anindyapal/desktop-cognition/blob/main/setup.sh"
    click PJ2 "https://github.com/anindyapal/desktop-cognition/blob/main/package.json"
    click REQ "https://github.com/anindyapal/desktop-cognition/blob/main/requirements.txt"
    click PT "https://github.com/anindyapal/desktop-cognition/blob/main/prompt_general.txt"
    click AS "https://github.com/anindyapal/desktop-cognition/blob/main/assets/icon.icns"

    %% Styles
    classDef electron fill:#D0E8FF,stroke:#0055AA,color:#003366
    classDef python fill:#E0FFE0,stroke:#00AA00,color:#006600
    classDef native fill:#FFE8D0,stroke:#AA5500,color:#663300
    classDef storage fill:#FFF0B3,stroke:#CCAA00,color:#664400
    classDef external fill:#F0F0F0,stroke:#888888,color:#555555
```
