# itwhizharsh/haki architecture

> Architecture diagram of the GitHub repository itwhizharsh/haki, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/itwhizharsh/haki
- Repository: https://github.com/itwhizharsh/haki

## Overview

1. **Top-level shape**
   - The repo is split into two runtime domains: **`HAKI.app`** in Swift and **`haki_core_service`** in Python.
   - This is not a single in-process app; it is a **two-process local assistant** with a hard boundary between UI/device access and AI orchestration.

2. **Process boundary and roles**
   - **Swift “Body” (`HAKI/`)** owns macOS-facing capabilities: permissions, audio, screen/capture, notifications, keyboard/menu-bar UI, local storage, and OS automation.
   - **Python “Mind” (`Core/`)** owns the assistant logic: orchestration, intent routing, dialogue, planning, memory/RAG, learning, and model backend abstraction.
   - The split is important for architecture diagrams because most side effects happen in the Swift process, while decisions and reasoning happen in Python.

3. **IPC layer**
   - The two processes communicate over a **UNIX domain socket** at `~/Library/Application Support/HAKI/haki_core.sock`.
   - The contract is defined in `proto/haki_ipc.proto` and implemented with generated gRPC stubs in `Core/core/ipc/proto/`.
   - Swift has its own IPC client (`IPCClient.swift`, `JSONIPCClient.swift`), so the transport is a key integration boundary rather than an internal detail.

4. **Swift app shell**
   - The Swift package starts from `main.swift` and `AppDelegate.swift`, with `CoreProcessManager.swift` responsible for launching/managing the Python service lifecycle.
   - `MenuBarUI.swift` and `UIState.swift` indicate the user experience is menu-bar driven, not a full windowed app.
   - `HAKI.entitlements` and the permission subsystem reflect that this process is the one interacting with macOS security-sensitive APIs.

5. **Device and OS-facing subsystems in Swift**
   - **Audio**: `AudioEngine`, `VAD`, `STTService`, `TTSService`, `VoiceEngine`, and `ClauseSegmenter` form the voice interaction path.
   - **Capture**: `ScreenReader` and `VisionOCR` handle screen text/image capture.
   - **OS actions**: `AppleScriptBridge`, `EventKitBridge`, and `NotificationManager` perform real system-side effects.
   - **Permissions**: `PermissionManager` and `HAKIPermission` gate access to macOS capabilities like Accessibility and screen/audio capture.

6. **Local persistence and user data in Swift**
   - `AppStore.swift` and `KeychainStore.swift` show the body process also owns durable local state and secrets handling.
   - The README’s privacy model matches this: sensitive local data stays on-device, and the Swift side appears to be the trusted boundary for encrypted storage and keychain-backed credentials.

7. **Core orchestration in Python**
   - `orchestrator.py` and `intent_router.py` are the main entry points for user-turn processing.
   - This layer likely receives events or utterances from the Swift process, routes them to the appropriate subsystem, and manages cancellation/turn control.
   - `dialogue_manager.py` handles clarification and slot-filling behavior when the user request is underspecified.

8. **Planning, execution, and safety**
   - `planner.py` plus `execution_engine.py` indicate a two-step flow: convert intent into a command plan, then execute it.
   - `safety_gate.py` is a distinct boundary before execution, which is important for diagramming because it separates “decision” from “allowed action.”
   - In the overall architecture, this is the point where AI-generated intent becomes constrained, checked behavior.

9. **Memory and learning pipeline**
   - `memory/` is a full subsystem: `vault.py`, `indexer.py`, `chunker.py`, `serializer.py`, `models.py`, and `store_result.py`.
   - This suggests a RAG-style pipeline: ingest notes/events, chunk and index them, persist them in a vault, and retrieve them for dialogue/orchestration.
   - `learning_engine.py` sits on top of this, extracting knowledge from conversations and deciding when a conversation has ended or when new memory should be stored.

10. **Model abstraction and multimodal AI**
   - `model_provider.py` abstracts backend access for **STT, LLM, TTS, and embeddings**.
   - That means the core logic is not tied to one vendor; the provider layer is the integration boundary for local or remote model backends.
   - `language_engine.py`, `persona_engine.py`, `mood_detector.py`, and `clock.py` provide higher-level context signals that shape responses and behavior.

11. **How data moves through the system**
   - Typical flow is: **user input / audio / screen context in Swift → IPC → Python orchestration/planning/memory/model calls → plan or response back over IPC → Swift performs UI/audio/OS actions**.
   - In the opposite direction, Swift can also send sensory context or events into the Core process to support memory, dialogue, or safety decisions.
   - This makes the architecture a feedback loop between “capture/action” and “reasoning.”

12. **Testing and repository structure**
   - The `Core/tests/` and `HAKI/Tests/` trees show each process is tested independently, with additional integration coverage around IPC and memory.
   - That separation mirrors the runtime split, so when drawing the architecture, it helps to place tests alongside the subsystem they validate rather than as a single shared layer.
   - The `.kiro/specs/` folder is design/specification material, not runtime architecture, but it documents the intended behavior of the same two-process system.

Main entry points: [`Core/core/orchestrator/orchestrator.py`](https://github.com/itwhizharsh/haki/blob/main/Core/core/orchestrator/orchestrator.py) (Orchestrator), [`HAKI/Sources/Subsystems/IPC/IPCClient.swift`](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/IPC/IPCClient.swift) (IPC Client), [`HAKI/Sources/HAKI/AppDelegate.swift`](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/HAKI/AppDelegate.swift) (Delegate), [`HAKI/Sources/Subsystems/Permissions/PermissionManager.swift`](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/Permissions/PermissionManager.swift) (Permissions)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_swift["HAKI.app"]
  node_swift_app_shell["App Shell<br/>swift entrypoint<br/>[main.swift]"]
  node_swift_app_delegate["Delegate<br/>app lifecycle<br/>[AppDelegate.swift]"]
  node_core_process_manager["Core Manager<br/>process supervisor"]
  node_menu_bar_ui["Menu Bar<br/>ui shell<br/>[MenuBarUI.swift]"]
  node_permission_manager["Permissions<br/>security gate"]
  node_audio_stack["Voice Stack<br/>audio pipeline<br/>[AudioEngine.swift]"]
  node_capture_stack["Capture<br/>[ScreenReader.swift]"]
  node_os_actions["OS Actions<br/>system automation"]
  node_swift_store[("Local Store<br/>persistent storage<br/>[AppStore.swift]")]
  node_swift_ipc["IPC Client<br/>transport client<br/>[IPCClient.swift]"]
end

subgraph group_python["haki_core_service"]
  node_core_service["Service Main<br/>python entrypoint"]
  node_ipc_server["IPC Server<br/>grpc server<br/>[server.py]"]
  node_orchestrator["Orchestrator<br/>turn manager<br/>[orchestrator.py]"]
  node_intent_router["Intent Router<br/>routing<br/>[intent_router.py]"]
  node_dialogue_manager["Dialogue<br/>clarification"]
  node_planner["Planner<br/>planning<br/>[planner.py]"]
  node_safety_gate{{"Safety Gate<br/>policy check<br/>[safety_gate.py]"}}
  node_execution_engine["Executor<br/>action runner"]
  node_memory_brain[("Memory<br/>rag subsystem<br/>[memory_brain.py]")]
  node_learning_engine["Learning<br/>knowledge extraction<br/>[learning_engine.py]"]
  node_model_provider["Model Provider<br/>backend adapter<br/>[model_provider.py]"]
  node_language_engine["Language<br/>response shaping<br/>[language_engine.py]"]
  node_persona_engine["Persona<br/>style engine<br/>[persona_engine.py]"]
  node_mood_detector["Mood<br/>context signal<br/>[mood_detector.py]"]
  node_clock["Clock<br/>context signal<br/>[clock.py]"]
end

subgraph group_ipc["IPC contract"]
  node_ipc_proto["IPC Proto<br/>gRPC schema<br/>[haki_ipc.proto]"]
end

node_swift_app_shell -->|"boots"| node_swift_app_delegate
node_swift_app_delegate -->|"starts core"| node_core_process_manager
node_swift_app_delegate -->|"wires UI"| node_menu_bar_ui
node_swift_app_delegate -->|"checks access"| node_permission_manager
node_permission_manager -->|"unlocks"| node_audio_stack
node_permission_manager -->|"unlocks"| node_capture_stack
node_swift_store -->|"state"| node_swift_app_delegate
node_audio_stack -->|"sends audio"| node_swift_ipc
node_capture_stack -->|"sends context"| node_swift_ipc
node_os_actions -->|"reports effects"| node_swift_ipc
node_menu_bar_ui -->|"user turn"| node_swift_ipc
node_swift_ipc -->|"uses schema"| node_ipc_proto
node_ipc_server -->|"implements schema"| node_ipc_proto
node_core_process_manager -->|"launches"| node_core_service
node_core_service -->|"serves"| node_ipc_server
node_ipc_server -->|"dispatches"| node_orchestrator
node_orchestrator -->|"routes"| node_intent_router
node_orchestrator -->|"clarifies"| node_dialogue_manager
node_intent_router -->|"plans"| node_planner
node_planner -->|"validates"| node_safety_gate
node_safety_gate -->|"approves"| node_execution_engine
node_orchestrator -->|"retrieves"| node_memory_brain
node_learning_engine -->|"stores"| node_memory_brain
node_orchestrator -->|"learns"| node_learning_engine
node_orchestrator -->|"invokes models"| node_model_provider
node_dialogue_manager -->|"shapes reply"| node_language_engine
node_orchestrator -->|"styles voice"| node_persona_engine
node_orchestrator -->|"reads mood"| node_mood_detector
node_orchestrator -->|"reads time"| node_clock
node_execution_engine -->|"returns actions"| node_swift_ipc

click node_swift_app_shell "https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/HAKI/main.swift"
click node_swift_app_delegate "https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/HAKI/AppDelegate.swift"
click node_core_process_manager "https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/HAKI/CoreProcessManager.swift"
click node_menu_bar_ui "https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/UI/MenuBarUI.swift"
click node_permission_manager "https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/Permissions/PermissionManager.swift"
click node_audio_stack "https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/Audio/AudioEngine.swift"
click node_capture_stack "https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/Capture/ScreenReader.swift"
click node_os_actions "https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/OSActions/AppleScriptBridge.swift"
click node_swift_store "https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/Store/AppStore.swift"
click node_swift_ipc "https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/IPC/IPCClient.swift"
click node_ipc_proto "https://github.com/itwhizharsh/haki/blob/main/proto/haki_ipc.proto"
click node_core_service "https://github.com/itwhizharsh/haki/blob/main/Core/haki_core_service.py"
click node_ipc_server "https://github.com/itwhizharsh/haki/blob/main/Core/core/ipc/server.py"
click node_orchestrator "https://github.com/itwhizharsh/haki/blob/main/Core/core/orchestrator/orchestrator.py"
click node_intent_router "https://github.com/itwhizharsh/haki/blob/main/Core/core/orchestrator/intent_router.py"
click node_dialogue_manager "https://github.com/itwhizharsh/haki/blob/main/Core/core/dialogue/dialogue_manager.py"
click node_planner "https://github.com/itwhizharsh/haki/blob/main/Core/core/planner/planner.py"
click node_safety_gate "https://github.com/itwhizharsh/haki/blob/main/Core/core/execution/safety_gate.py"
click node_execution_engine "https://github.com/itwhizharsh/haki/blob/main/Core/core/execution/execution_engine.py"
click node_memory_brain "https://github.com/itwhizharsh/haki/blob/main/Core/core/memory/memory_brain.py"
click node_learning_engine "https://github.com/itwhizharsh/haki/blob/main/Core/core/learning/learning_engine.py"
click node_model_provider "https://github.com/itwhizharsh/haki/blob/main/Core/core/model_provider/model_provider.py"
click node_language_engine "https://github.com/itwhizharsh/haki/blob/main/Core/core/language/language_engine.py"
click node_persona_engine "https://github.com/itwhizharsh/haki/blob/main/Core/core/persona/persona_engine.py"
click node_mood_detector "https://github.com/itwhizharsh/haki/blob/main/Core/core/mood/mood_detector.py"
click node_clock "https://github.com/itwhizharsh/haki/blob/main/Core/core/clock/clock.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_swift_app_shell,node_swift_app_delegate,node_core_process_manager,node_menu_bar_ui,node_permission_manager,node_audio_stack,node_capture_stack,node_os_actions,node_swift_store,node_swift_ipc toneBlue
class node_core_service,node_ipc_server,node_orchestrator,node_intent_router,node_dialogue_manager,node_planner,node_safety_gate,node_execution_engine,node_memory_brain,node_learning_engine,node_model_provider,node_language_engine,node_persona_engine,node_mood_detector,node_clock toneAmber
class node_ipc_proto toneMint
```

## Components

### HAKI.app

macOS-facing body process for capture, actions, UI, and local storage

- [App Shell](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/HAKI/main.swift): swift entrypoint · Starts the macOS app and manages the core service lifecycle · `HAKI/Sources/HAKI/main.swift`
- [Delegate](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/HAKI/AppDelegate.swift): app lifecycle · Coordinates app startup and runtime wiring · `HAKI/Sources/HAKI/AppDelegate.swift`
- [Core Manager](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/HAKI/CoreProcessManager.swift): process supervisor · Launches and supervises the Python assistant service · `HAKI/Sources/HAKI/CoreProcessManager.swift`
- [Menu Bar](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/UI/MenuBarUI.swift): ui shell · Menu-bar driven user interface and status surface · `HAKI/Sources/Subsystems/UI/MenuBarUI.swift`
- [Permissions](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/Permissions/PermissionManager.swift): security gate · Gates macOS accessibility, screen, and audio capabilities · `HAKI/Sources/Subsystems/Permissions/PermissionManager.swift`
- [Voice Stack](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/Audio/AudioEngine.swift): audio pipeline · Audio capture, VAD, STT, TTS, and voice control · `HAKI/Sources/Subsystems/Audio/AudioEngine.swift`
- [Capture](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/Capture/ScreenReader.swift): screen capture · Reads screen content and OCR text for assistant context · `HAKI/Sources/Subsystems/Capture/ScreenReader.swift`
- [OS Actions](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/OSActions/AppleScriptBridge.swift): system automation · Performs AppleScript, calendar, and notification side effects · `HAKI/Sources/Subsystems/OSActions/AppleScriptBridge.swift`
- [Local Store](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/Store/AppStore.swift): persistent storage · Owns app state and keychain-backed secrets · `HAKI/Sources/Subsystems/Store/AppStore.swift`
- [IPC Client](https://github.com/itwhizharsh/haki/blob/main/HAKI/Sources/Subsystems/IPC/IPCClient.swift): transport client · Swift client for Python service communication · `HAKI/Sources/Subsystems/IPC/IPCClient.swift`

### haki_core_service

Python mind process for orchestration, planning, memory, and model access

- [Service Main](https://github.com/itwhizharsh/haki/blob/main/Core/haki_core_service.py): python entrypoint · Starts the Python core service and server loop · `Core/haki_core_service.py`
- [IPC Server](https://github.com/itwhizharsh/haki/blob/main/Core/core/ipc/server.py): grpc server · Receives Swift requests and streams responses · `Core/core/ipc/server.py`
- [Orchestrator](https://github.com/itwhizharsh/haki/blob/main/Core/core/orchestrator/orchestrator.py): turn manager · Coordinates user-turn processing across core subsystems · `Core/core/orchestrator/orchestrator.py`
- [Intent Router](https://github.com/itwhizharsh/haki/blob/main/Core/core/orchestrator/intent_router.py): routing · Maps requests to specialized assistant behavior · `Core/core/orchestrator/intent_router.py`
- [Dialogue](https://github.com/itwhizharsh/haki/blob/main/Core/core/dialogue/dialogue_manager.py): clarification · Handles slot-filling and conversational repair · `Core/core/dialogue/dialogue_manager.py`
- [Planner](https://github.com/itwhizharsh/haki/blob/main/Core/core/planner/planner.py): planning · Converts intent into structured action plans · `Core/core/planner/planner.py`
- [Safety Gate](https://github.com/itwhizharsh/haki/blob/main/Core/core/execution/safety_gate.py): policy check · Constrains plans before execution · `Core/core/execution/safety_gate.py`
- [Executor](https://github.com/itwhizharsh/haki/blob/main/Core/core/execution/execution_engine.py): action runner · Runs approved plans and returns outcomes · `Core/core/execution/execution_engine.py`
- [Memory](https://github.com/itwhizharsh/haki/blob/main/Core/core/memory/memory_brain.py): rag subsystem · Coordinates vault, indexing, chunking, and retrieval · `Core/core/memory/memory_brain.py`
- [Learning](https://github.com/itwhizharsh/haki/blob/main/Core/core/learning/learning_engine.py): knowledge extraction · Captures durable knowledge from conversations · `Core/core/learning/learning_engine.py`
- [Model Provider](https://github.com/itwhizharsh/haki/blob/main/Core/core/model_provider/model_provider.py): backend adapter · Abstracts STT, LLM, TTS, and embedding backends · `Core/core/model_provider/model_provider.py`
- [Language](https://github.com/itwhizharsh/haki/blob/main/Core/core/language/language_engine.py): response shaping · Provides higher-level language behavior signals · `Core/core/language/language_engine.py`
- [Persona](https://github.com/itwhizharsh/haki/blob/main/Core/core/persona/persona_engine.py): style engine · Shapes assistant voice and persona · `Core/core/persona/persona_engine.py`
- [Mood](https://github.com/itwhizharsh/haki/blob/main/Core/core/mood/mood_detector.py): context signal · Infers user mood and conversational tone · `Core/core/mood/mood_detector.py`
- [Clock](https://github.com/itwhizharsh/haki/blob/main/Core/core/clock/clock.py): context signal · Supplies time-aware context · `Core/core/clock/clock.py`

### IPC contract

Shared transport and schema boundary between Swift and Python

- [IPC Proto](https://github.com/itwhizharsh/haki/blob/main/proto/haki_ipc.proto): gRPC schema · Shared IPC contract and generated stubs · `proto/haki_ipc.proto`

## Connections

- App Shell → Delegate: boots · App startup flows into lifecycle coordination
- Delegate → Core Manager: starts core · Delegate asks manager to launch and supervise Python service
- Delegate → Menu Bar: wires UI · Initializes the menu-bar interface
- Delegate → Permissions: checks access · Ensures required macOS permissions are available
- Permissions → Voice Stack: unlocks · Gates audio and speech features
- Permissions → Capture: unlocks · Gates screen capture and OCR
- Local Store → Delegate: state · Provides persisted app state and secrets
- Voice Stack → IPC Client: sends audio · Transmits recognized speech or audio events to core
- Capture → IPC Client: sends context · Sends screen-derived context to core
- OS Actions → IPC Client: reports effects · Receives commands or returns operation results through IPC
- Menu Bar → IPC Client: user turn · User interactions are forwarded to the core process
- IPC Client → IPC Proto: uses schema · Client follows the shared IPC contract
- IPC Server → IPC Proto: implements schema · Server implements the same gRPC contract
- Core Manager → Service Main: launches · Supervisor starts the Python service process
- Service Main → IPC Server: serves · Service exposes the IPC server
- IPC Server → Orchestrator: dispatches · Incoming requests enter the orchestration layer
- Orchestrator → Intent Router: routes · Orchestrator selects the right handling path
- Orchestrator → Dialogue: clarifies · Underspecified turns are repaired through dialogue
- Intent Router → Planner: plans · Intent becomes a structured plan
- Planner → Safety Gate: validates · Plans are checked before execution
- Safety Gate → Executor: approves · Only safe actions proceed to execution
- Orchestrator → Memory: retrieves · Turn processing consults memory and context
- Learning → Memory: stores · Learnt knowledge is written back into memory
- Orchestrator → Learning: learns · Conversation outcomes feed learning
- Orchestrator → Model Provider: invokes models · Uses model backends for reasoning and generation
- Dialogue → Language: shapes reply · Dialogue behavior is informed by language utilities
- Orchestrator → Persona: styles voice · Persona influences the final assistant response
- Orchestrator → Mood: reads mood · Mood signal informs interpretation and tone
- Orchestrator → Clock: reads time · Time context can affect decisions and phrasing
- Executor → IPC Client: returns actions · Approved actions and results flow back to Swift