itwhizharsh/haki

1. Top-level shape

- The repo is split into two runtime domains: HAKI.app in Swift and haki_core_service in Python.

- This is not a single in-process app; it is a two-process local assistant with a hard boundary between UI/device access and AI orchestration.

2. Process boundary and roles

- Swift “Body” (HAKI/) owns macOS-facing capabilities: permissions, audio, screen/capture, notifications, keyboard/menu-bar UI, local storage, and OS automation.

- Python “Mind” (Core/) owns the assistant logic: orchestration, intent routing, dialogue, planning, memory/RAG, learning, and model backend abstraction.

- The split is important for architecture diagrams because most side effects happen in the Swift process, while decisions and reasoning happen in Python.

3. IPC layer

- The two processes communicate over a UNIX domain socket at ~/Library/Application Support/HAKI/haki_core.sock.

- The contract is defined in proto/haki_ipc.proto and implemented with generated gRPC stubs in Core/core/ipc/proto/.

- Swift has its own IPC client (IPCClient.swift, JSONIPCClient.swift), so the transport is a key integration boundary rather than an internal detail.

4. Swift app shell

- The Swift package starts from main.swift and AppDelegate.swift, with CoreProcessManager.swift responsible for launching/managing the Python service lifecycle.

- MenuBarUI.swift and UIState.swift indicate the user experience is menu-bar driven, not a full windowed app.

- HAKI.entitlements and the permission subsystem reflect that this process is the one interacting with macOS security-sensitive APIs.

5. Device and OS-facing subsystems in Swift

- Audio: AudioEngine, VAD, STTService, TTSService, VoiceEngine, and ClauseSegmenter form the voice interaction path.

- Capture: ScreenReader and VisionOCR handle screen text/image capture.

- OS actions: AppleScriptBridge, EventKitBridge, and NotificationManager perform real system-side effects.

- Permissions: PermissionManager and HAKIPermission gate access to macOS capabilities like Accessibility and screen/audio capture.

6. Local persistence and user data in Swift

- AppStore.swift and KeychainStore.swift show the body process also owns durable local state and secrets handling.

- The README’s privacy model matches this: sensitive local data stays on-device, and the Swift side appears to be the trusted boundary for encrypted storage and keychain-backed credentials.

7. Core orchestration in Python

- orchestrator.py and intent_router.py are the main entry points for user-turn processing.

- This layer likely receives events or utterances from the Swift process, routes them to the appropriate subsystem, and manages cancellation/turn control.

- dialogue_manager.py handles clarification and slot-filling behavior when the user request is underspecified.

8. Planning, execution, and safety

- planner.py plus execution_engine.py indicate a two-step flow: convert intent into a command plan, then execute it.

- safety_gate.py is a distinct boundary before execution, which is important for diagramming because it separates “decision” from “allowed action.”

- In the overall architecture, this is the point where AI-generated intent becomes constrained, checked behavior.

9. Memory and learning pipeline

- memory/ is a full subsystem: vault.py, indexer.py, chunker.py, serializer.py, models.py, and store_result.py.

- This suggests a RAG-style pipeline: ingest notes/events, chunk and index them, persist them in a vault, and retrieve them for dialogue/orchestration.

- learning_engine.py sits on top of this, extracting knowledge from conversations and deciding when a conversation has ended or when new memory should be stored.

10. Model abstraction and multimodal AI

- model_provider.py abstracts backend access for STT, LLM, TTS, and embeddings.

- That means the core logic is not tied to one vendor; the provider layer is the integration boundary for local or remote model backends.

- language_engine.py, persona_engine.py, mood_detector.py, and clock.py provide higher-level context signals that shape responses and behavior.

11. How data moves through the system

- Typical flow is: user input / audio / screen context in Swift → IPC → Python orchestration/planning/memory/model calls → plan or response back over IPC → Swift performs UI/audio/OS actions.

- In the opposite direction, Swift can also send sensory context or events into the Core process to support memory, dialogue, or safety decisions.

- This makes the architecture a feedback loop between “capture/action” and “reasoning.”

12. Testing and repository structure

- The Core/tests/ and HAKI/Tests/ trees show each process is tested independently, with additional integration coverage around IPC and memory.

- That separation mirrors the runtime split, so when drawing the architecture, it helps to place tests alongside the subsystem they validate rather than as a single shared layer.

- The .kiro/specs/ folder is design/specification material, not runtime architecture, but it documents the intended behavior of the same two-process system.