# lfallardi/gemini-mac-pilot-lf architecture

> Architecture diagram of the GitHub repository lfallardi/gemini-mac-pilot-lf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lfallardi/gemini-mac-pilot-lf
- Repository: https://github.com/lfallardi/gemini-mac-pilot-lf

## Overview

Gemini Mac Pilot turns user requests into macOS, browser, and Google Workspace actions. In the primary voice workflow, audio reaches the Gemini Live voice layer, which sends an execute_task request to the brain; the brain selects and dispatches tools, then returns a result. Users can also submit text through the overlay or CLI. The overlay receives task and progress events over WebSocket. Tool implementations and workspace integrations are partly unsampled, so their internal dependencies are shown only where the README or source index supports them.

Main entry points: [`mac_pilot/brain.py`](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/brain.py) (Task brain), [`mac_pilot/tools/schema.py`](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/schema.py) (Tool declarations), [`mac_pilot/voice.py`](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/voice.py) (Voice session), [`main.py`](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/main.py) (Runtime entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User Entry"]
  node_main["Runtime entry<br/>[main.py]"]
  node_overlay["Floating overlay<br/>[app.py]"]
  node_frontend["Overlay frontend<br/>[app.js]"]
  node_cloud["Cloud API<br/>[cloud_api.py]"]
end

subgraph group_reasoning["Voice and Reasoning"]
  node_brain["Task brain<br/>[brain.py]"]
  node_voice["Voice session<br/>[voice.py]"]
  node_prompts["Agent prompts<br/>[prompts.py]"]
  node_config["Model config<br/>[config.py]"]
  node_tool_schema["Tool declarations<br/>[schema.py]"]
end

subgraph group_native["macOS Tools"]
  node_native_ui["Accessibility control<br/>[accessibility.py]"]
  node_apps["App control<br/>[apps.py]"]
  node_keyboard["Keyboard control<br/>[keyboard.py]"]
  node_shell["Shell actions<br/>[shell.py]"]
end

subgraph group_web["Browser and Workspace"]
  node_browser["Browser automation<br/>[browser.py]"]
  node_workspace["Workspace actions<br/>[workspace.py]"]
end

subgraph group_interface["UI and Events"]
  node_events["Event state<br/>[events.py]"]
  node_server["WebSocket server<br/>[server.py]"]
  node_setup["Setup status<br/>[setup_check.py]"]
end

node_user(("Mac user"))
node_gemini["Vertex AI Gemini"]

node_user -->|"submits task"| node_main
node_main -->|"starts voice"| node_voice
node_main -->|"starts server"| node_server
node_main -->|"opens overlay"| node_overlay
node_user -->|"speaks or types"| node_overlay
node_overlay -.->|"displays UI"| node_frontend
node_frontend -->|"sends commands"| node_server
node_server -->|"queues text"| node_main
node_main -->|"runs task"| node_brain
node_voice -->|"runs task"| node_brain
node_voice -->|"uses model config"| node_config
node_brain -->|"uses model client"| node_config
node_brain -->|"uses instructions"| node_prompts
node_voice -->|"uses instructions"| node_prompts
node_config -->|"configures clients"| node_gemini
node_brain -->|"requests decisions"| node_gemini
node_voice -->|"streams audio"| node_gemini
node_brain -->|"dispatches functions"| node_tool_schema
node_tool_schema -->|"dispatches controls"| node_native_ui
node_tool_schema -->|"dispatches app tools"| node_apps
node_tool_schema -->|"dispatches keys"| node_keyboard
node_tool_schema -->|"dispatches commands"| node_shell
node_tool_schema -->|"dispatches browser tools"| node_browser
node_tool_schema -->|"dispatches workspace tools"| node_workspace
node_apps -->|"reads app state"| node_native_ui
node_keyboard -->|"refreshes UI"| node_native_ui
node_brain -->|"publishes task progress"| node_events
node_voice -->|"publishes voice state"| node_events
node_server -->|"subscribes events"| node_events
node_server -->|"reads setup status"| node_setup
node_cloud -->|"calls brain"| node_brain

click node_main "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/main.py"
click node_overlay "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/ui/app.py"
click node_frontend "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/ui/static/app.js"
click node_brain "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/brain.py"
click node_voice "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/voice.py"
click node_prompts "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/prompts.py"
click node_config "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/config.py"
click node_tool_schema "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/schema.py"
click node_native_ui "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/accessibility.py"
click node_apps "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/apps.py"
click node_keyboard "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/keyboard.py"
click node_shell "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/shell.py"
click node_browser "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/browser.py"
click node_workspace "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/workspace.py"
click node_events "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/events.py"
click node_server "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/ui/server.py"
click node_setup "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/setup_check.py"
click node_cloud "https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/cloud_api.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_overlay,node_frontend,node_cloud,node_user toneBlue
class node_brain,node_voice,node_prompts,node_config,node_tool_schema toneAmber
class node_native_ui,node_apps,node_keyboard,node_shell toneMint
class node_browser,node_workspace toneRose
class node_events,node_server,node_setup,node_gemini toneIndigo
```

## Components

### User Entry

- [Runtime entry](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/main.py): `main.py`
- [Floating overlay](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/ui/app.py): `mac_pilot/ui/app.py`
- [Overlay frontend](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/ui/static/app.js): `mac_pilot/ui/static/app.js`
- [Cloud API](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/cloud_api.py): `cloud_api.py`

### Voice and Reasoning

- [Task brain](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/brain.py): `mac_pilot/brain.py`
- [Voice session](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/voice.py): `mac_pilot/voice.py`
- [Agent prompts](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/prompts.py): `mac_pilot/prompts.py`
- [Model config](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/config.py): `mac_pilot/config.py`
- [Tool declarations](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/schema.py): `mac_pilot/tools/schema.py`

### macOS Tools

- [Accessibility control](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/accessibility.py): `mac_pilot/tools/accessibility.py`
- [App control](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/apps.py): `mac_pilot/tools/apps.py`
- [Keyboard control](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/keyboard.py): `mac_pilot/tools/keyboard.py`
- [Shell actions](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/shell.py): `mac_pilot/tools/shell.py`

### Browser and Workspace

- [Browser automation](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/browser.py): `mac_pilot/tools/browser.py`
- [Workspace actions](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/tools/workspace.py): `mac_pilot/tools/workspace.py`

### UI and Events

- [Event state](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/events.py): `mac_pilot/events.py`
- [WebSocket server](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/ui/server.py): `mac_pilot/ui/server.py`
- [Setup status](https://github.com/lfallardi/gemini-mac-pilot-lf/blob/main/mac_pilot/setup_check.py): `mac_pilot/setup_check.py`

### Other components

- **Mac user**
- **Vertex AI Gemini**

## Connections

- Mac user → Runtime entry: submits task (evidence: `README.md`)
- Runtime entry → Voice session: starts voice (evidence: `main.py`)
- Runtime entry → WebSocket server: starts server (evidence: `main.py`)
- Runtime entry → Floating overlay: opens overlay (evidence: `main.py`)
- Mac user → Floating overlay: speaks or types (evidence: `README.md`)
- Floating overlay → Overlay frontend: displays UI
- Overlay frontend → WebSocket server: sends commands (evidence: `README.md`)
- WebSocket server → Runtime entry: queues text (evidence: `mac_pilot/ui/server.py`)
- Runtime entry → Task brain: runs task (evidence: `main.py`)
- Voice session → Task brain: runs task (evidence: `mac_pilot/voice.py`)
- Voice session → Model config: uses model config (evidence: `mac_pilot/voice.py`)
- Task brain → Model config: uses model client (evidence: `mac_pilot/brain.py`)
- Task brain → Agent prompts: uses instructions (evidence: `mac_pilot/brain.py`)
- Voice session → Agent prompts: uses instructions (evidence: `mac_pilot/voice.py`)
- Model config → Vertex AI Gemini: configures clients (evidence: `mac_pilot/config.py`)
- Task brain → Vertex AI Gemini: requests decisions (evidence: `README.md`)
- Voice session → Vertex AI Gemini: streams audio (evidence: `README.md`)
- Task brain → Tool declarations: dispatches functions (evidence: `mac_pilot/brain.py`)
- Tool declarations → Accessibility control: dispatches controls (evidence: `mac_pilot/tools/schema.py`)
- Tool declarations → App control: dispatches app tools (evidence: `mac_pilot/tools/schema.py`)
- Tool declarations → Keyboard control: dispatches keys (evidence: `mac_pilot/tools/schema.py`)
- Tool declarations → Shell actions: dispatches commands (evidence: `mac_pilot/tools/schema.py`)
- Tool declarations → Browser automation: dispatches browser tools (evidence: `mac_pilot/tools/schema.py`)
- Tool declarations → Workspace actions: dispatches workspace tools (evidence: `mac_pilot/tools/schema.py`)
- App control → Accessibility control: reads app state (evidence: `mac_pilot/tools/apps.py`)
- Keyboard control → Accessibility control: refreshes UI (evidence: `mac_pilot/tools/keyboard.py`)
- Task brain → Event state: publishes task progress (evidence: `mac_pilot/brain.py`)
- Voice session → Event state: publishes voice state (evidence: `mac_pilot/voice.py`)
- WebSocket server → Event state: subscribes events (evidence: `mac_pilot/ui/server.py`)
- WebSocket server → Setup status: reads setup status (evidence: `mac_pilot/ui/server.py`)
- Cloud API → Task brain: calls brain (evidence: `cloud_api.py`)
