# typhoidcoma/patyna architecture

> Architecture diagram of the GitHub repository typhoidcoma/patyna, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/typhoidcoma/patyna
- Repository: https://github.com/typhoidcoma/patyna

## Overview

1. **Overall shape**
This repo is a browser-based, multi-entry application built with Vite + TypeScript. It ships three user-facing experiences from the same codebase: the main real-time avatar app (`/`), a text-only P0 demo (`/demo.html`), and the more polished LUMINORA dashboard (`/demo2.html`). The architecture is centered on a shared realtime “avatar + conversation” core, with two separate dashboard/product shells layered on top.

2. **Entry points and orchestration**
`src/main.ts` and `src/app.ts` are the main app bootstrap/orchestrator path. The two demos have their own bootstraps and app/state modules under `src/demo/` and `src/demo2/`. This separation is important: the realtime avatar engine is shared conceptually, but each experience owns its own UI layout, state model, and user flow.

3. **Core runtime model**
`src/core/` provides the app-wide coordination primitives: a typed event bus and a finite state machine that drives the interaction states (`idle`, `listening`, `thinking`, `speaking`). This core layer sits above media, scene, and backend code and acts as the glue between user input, network responses, and avatar animation.

4. **Real-time voice pipeline**
The voice stack is split across `src/voice/` and `src/audio/`. Voice input uses voice activity detection plus a speech-to-text provider abstraction, with a Web Speech API implementation in `web-speech-stt.ts`. Voice output uses `ElevenLabs` streaming TTS over WebSocket, then routes PCM into an `AudioWorklet`-backed playback path via `tts-player.ts` and `audio-manager.ts`. The audio output is also analyzed for amplitude so the avatar can animate in sync with speech.

5. **3D avatar and visual scene**
`src/scene/` contains the Three.js rendering layer: scene setup, avatar mesh, environment/background, mood-driven animation profiles, and color mapping. `avatar-controller.ts` bridges tracking data into avatar motion, while `scene-manager.ts` owns rendering lifecycle and camera/lights. This layer is purely presentation-focused and consumes higher-level state such as speaking/mood/presence.

6. **Camera, face tracking, and presence**
`src/tracking/` handles webcam access, MediaPipe face landmarks, and presence detection. Presence is derived from face visibility over time and feeds the avatar’s dimming/eyes-closed behavior. This is a distinct subsystem boundary: camera input is not just for rendering, but also for presence semantics that affect the whole experience.

7. **Backend communication and memory**
`src/comm/` contains the WebSocket protocol, message codec, and reconnecting client used to talk to the Aelora backend. `src/api/aelora-client.ts` handles REST calls for users, sessions, memory, and mood. In practice, the backend is the app’s external brain: chat responses, user-scoped memory/vault data, and session state live there, while the browser maintains the interactive front end.

8. **Domain-specific quest/task logic**
`src/quests/` is the app’s domain translation layer for task-oriented interactions. It parses intents from free text, maps quests to tasks, subscribes/fetches quest data, and matches quests by title hints. This is the logic that turns conversational input or backend content into actionable dashboard items in the demo experiences.

9. **Product shells: P0 demo and LUMINORA**
`src/demo/` is a lightweight dashboard experience with seeded state, drag-and-drop widgets, and celebration effects for task completion. `src/demo2/` is the richer LUMINORA product shell: it has its own componentized dashboard, modal system, journal input, vault/memory view, weekly rhythm UI, and task/goal panels. These shells consume the shared backend/voice concepts but do not depend on the full 3D camera experience.

10. **UI, styling, and interaction plumbing**
`src/ui/` contains reusable HUD/sidebar widgets and drag behavior for floating panels; `src/demo2/components/` is a more structured component set for the LUMINORA dashboard. Global styling is split between shared styles in `src/styles/` and experience-specific CSS in `src/demo2/demo2.css` and `src/demo/demo-overrides.css`. This is a mostly client-side UI architecture with no separate frontend framework layer.

11. **External services and operational boundaries**
The app depends on ElevenLabs for streaming TTS, Aelora for chat/memory/session APIs, and optionally Supabase-related plumbing for auth or feedback (`src/auth/`, `src/api/feedback-client.ts`). The browser is the primary runtime, with a worker in `public/workers/` supporting audio playback. Static assets and HTML entrypoints are served by Vite, making this a multi-page browser application rather than a single SPA bundle.

12. **How data flows through the system**
Typical flow is: user input via mic/camera or text UI enters the core event/state system, which coordinates voice recognition, backend calls, and avatar state. Backend responses and TTS stream back into the audio subsystem, while the scene layer reacts to speaking amplitude, mood, and presence. In the dashboard modes, task and memory data flow from Aelora/local demo state into UI panels, with completion events triggering celebratory effects and modal updates.

Main entry points: [`src/app.ts`](https://github.com/typhoidcoma/patyna/blob/main/src/app.ts) (App orchestrator), [`src/demo2/demo2-main.ts`](https://github.com/typhoidcoma/patyna/blob/main/src/demo2/demo2-main.ts) (LUMINORA), [`src/voice/voice-manager.ts`](https://github.com/typhoidcoma/patyna/blob/main/src/voice/voice-manager.ts) (Voice manager), [`src/demo/demo-main.ts`](https://github.com/typhoidcoma/patyna/blob/main/src/demo/demo-main.ts) (P0 demo)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_runtime["Browser runtime"]
  node_node_main["Main app<br/>bootstrap<br/>[main.ts]"]
  node_node_app["App orchestrator<br/>[app.ts]"]
  node_node_core_bus(("Event bus<br/>coordination<br/>[event-bus.ts]"))
  node_node_core_state{{"State machine<br/>fsm<br/>[state-machine.ts]"}}
end

subgraph group_group_avatar["Avatar experience"]
  node_node_voice_mgr["Voice manager<br/>voice orchestration<br/>[voice-manager.ts]"]
  node_node_vad(("VAD<br/>speech detection<br/>[vad.ts]"))
  node_node_stt["STT provider<br/>speech-to-text<br/>[stt-provider.ts]"]
  node_node_web_speech["Web Speech STT<br/>browser STT<br/>[web-speech-stt.ts]"]
  node_node_tts["TTS stream<br/>text-to-speech<br/>[elevenlabs-tts.ts]"]
  node_node_tts_player["TTS player<br/>audio playback<br/>[tts-player.ts]"]
  node_node_audio_mgr["Audio manager<br/>playback<br/>[audio-manager.ts]"]
  node_node_scene_mgr["Scene manager<br/>rendering<br/>[scene-manager.ts]"]
  node_node_avatar_ctrl["Avatar controller<br/>motion bridge"]
  node_node_tracking["Face tracking<br/>vision<br/>[face-tracker.ts]"]
  node_node_presence(("Presence<br/>state derivation"))
end

subgraph group_group_backend["Backend integration"]
  node_node_ws_client["WebSocket client<br/>realtime transport"]
  node_node_codec["Message codec<br/>serialization<br/>[message-codec.ts]"]
  node_node_protocol["Protocol<br/>message types<br/>[protocol.ts]"]
  node_node_aelora["Aelora client<br/>REST client<br/>[aelora-client.ts]"]
  node_node_quests["Quest logic<br/>domain translation"]
end

subgraph group_group_products["Product shells"]
  node_node_demo["P0 demo<br/>demo shell<br/>[demo-main.ts]"]
  node_node_demo2["LUMINORA<br/>dashboard shell<br/>[demo2-main.ts]"]
  node_node_demo_ui["Demo UI<br/>demo components<br/>[demo-hud.ts]"]
  node_node_demo2_ui["Dashboard UI<br/>component set<br/>[modal-manager.ts]"]
  node_node_fx(("Celebration FX<br/>effects<br/>[celebration.ts]"))
end

node_node_main -->|"boots"| node_node_app
node_node_app -->|"publishes"| node_node_core_bus
node_node_app -->|"drives"| node_node_core_state
node_node_app -->|"wires"| node_node_voice_mgr
node_node_app -->|"wires"| node_node_scene_mgr
node_node_voice_mgr -->|"uses"| node_node_vad
node_node_voice_mgr -->|"selects"| node_node_stt
node_node_stt -->|"implements"| node_node_web_speech
node_node_voice_mgr -->|"streams"| node_node_tts
node_node_tts -->|"feeds"| node_node_tts_player
node_node_tts_player -->|"routes"| node_node_audio_mgr
node_node_audio_mgr -->|"analyzes"| node_node_scene_mgr
node_node_tracking -->|"derives"| node_node_presence
node_node_tracking -->|"feeds"| node_node_avatar_ctrl
node_node_presence -->|"modulates"| node_node_avatar_ctrl
node_node_scene_mgr -->|"hosts"| node_node_avatar_ctrl
node_node_ws_client -->|"encodes"| node_node_codec
node_node_codec -->|"conforms to"| node_node_protocol
node_node_aelora -->|"supplies data"| node_node_quests
node_node_demo -->|"composes"| node_node_demo_ui
node_node_demo2 -->|"composes"| node_node_demo2_ui
node_node_demo -->|"syncs"| node_node_aelora
node_node_demo2 -->|"syncs"| node_node_aelora
node_node_demo2 -->|"maps tasks"| node_node_quests
node_node_demo2 -->|"listens"| node_node_core_bus
node_node_demo -.->|"celebrates"| node_node_fx

click node_node_main "https://github.com/typhoidcoma/patyna/blob/main/src/main.ts"
click node_node_app "https://github.com/typhoidcoma/patyna/blob/main/src/app.ts"
click node_node_core_bus "https://github.com/typhoidcoma/patyna/blob/main/src/core/event-bus.ts"
click node_node_core_state "https://github.com/typhoidcoma/patyna/blob/main/src/core/state-machine.ts"
click node_node_voice_mgr "https://github.com/typhoidcoma/patyna/blob/main/src/voice/voice-manager.ts"
click node_node_vad "https://github.com/typhoidcoma/patyna/blob/main/src/voice/vad.ts"
click node_node_stt "https://github.com/typhoidcoma/patyna/blob/main/src/voice/stt-provider.ts"
click node_node_web_speech "https://github.com/typhoidcoma/patyna/blob/main/src/voice/web-speech-stt.ts"
click node_node_tts "https://github.com/typhoidcoma/patyna/blob/main/src/audio/elevenlabs-tts.ts"
click node_node_tts_player "https://github.com/typhoidcoma/patyna/blob/main/src/audio/tts-player.ts"
click node_node_audio_mgr "https://github.com/typhoidcoma/patyna/blob/main/src/audio/audio-manager.ts"
click node_node_scene_mgr "https://github.com/typhoidcoma/patyna/blob/main/src/scene/scene-manager.ts"
click node_node_avatar_ctrl "https://github.com/typhoidcoma/patyna/blob/main/src/scene/avatar-controller.ts"
click node_node_tracking "https://github.com/typhoidcoma/patyna/blob/main/src/tracking/face-tracker.ts"
click node_node_presence "https://github.com/typhoidcoma/patyna/blob/main/src/tracking/presence-manager.ts"
click node_node_ws_client "https://github.com/typhoidcoma/patyna/blob/main/src/comm/websocket-client.ts"
click node_node_codec "https://github.com/typhoidcoma/patyna/blob/main/src/comm/message-codec.ts"
click node_node_protocol "https://github.com/typhoidcoma/patyna/blob/main/src/comm/protocol.ts"
click node_node_aelora "https://github.com/typhoidcoma/patyna/blob/main/src/api/aelora-client.ts"
click node_node_quests "https://github.com/typhoidcoma/patyna/blob/main/src/quests/parse-add-task-intent.ts"
click node_node_demo "https://github.com/typhoidcoma/patyna/blob/main/src/demo/demo-main.ts"
click node_node_demo2 "https://github.com/typhoidcoma/patyna/blob/main/src/demo2/demo2-main.ts"
click node_node_demo_ui "https://github.com/typhoidcoma/patyna/blob/main/src/ui/demo-hud.ts"
click node_node_demo2_ui "https://github.com/typhoidcoma/patyna/blob/main/src/demo2/components/modal-manager.ts"
click node_node_fx "https://github.com/typhoidcoma/patyna/blob/main/src/fx/celebration.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_main,node_node_app,node_node_core_bus,node_node_core_state toneBlue
class node_node_voice_mgr,node_node_vad,node_node_stt,node_node_web_speech,node_node_tts,node_node_tts_player,node_node_audio_mgr,node_node_scene_mgr,node_node_avatar_ctrl,node_node_tracking,node_node_presence toneAmber
class node_node_ws_client,node_node_codec,node_node_protocol,node_node_aelora,node_node_quests toneMint
class node_node_demo,node_node_demo2,node_node_demo_ui,node_node_demo2_ui,node_node_fx toneRose
```

## Components

### Browser runtime

Shared bootstrap, app coordination, and cross-cutting state.

- [Main app](https://github.com/typhoidcoma/patyna/blob/main/src/main.ts): bootstrap · Primary browser entry for the real-time avatar experience. · `src/main.ts`
- [App orchestrator](https://github.com/typhoidcoma/patyna/blob/main/src/app.ts): orchestrator · Top-level application wiring and lifecycle coordination. · `src/app.ts`
- [Event bus](https://github.com/typhoidcoma/patyna/blob/main/src/core/event-bus.ts): coordination · Typed pub/sub channel for app-wide events. · `src/core/event-bus.ts`
- [State machine](https://github.com/typhoidcoma/patyna/blob/main/src/core/state-machine.ts): fsm · Interaction state control for idle, listening, thinking, and speaking. · `src/core/state-machine.ts`

### Avatar experience

Voice, tracking, audio, and 3D presentation for the real-time app.

- [Voice manager](https://github.com/typhoidcoma/patyna/blob/main/src/voice/voice-manager.ts): voice orchestration · Coordinates speech input, VAD, and provider selection. · `src/voice/voice-manager.ts`
- [VAD](https://github.com/typhoidcoma/patyna/blob/main/src/voice/vad.ts): speech detection · Detects active speech segments from microphone input. · `src/voice/vad.ts`
- [STT provider](https://github.com/typhoidcoma/patyna/blob/main/src/voice/stt-provider.ts): speech-to-text · Abstract speech recognition interface used by the voice stack. · `src/voice/stt-provider.ts`
- [Web Speech STT](https://github.com/typhoidcoma/patyna/blob/main/src/voice/web-speech-stt.ts): browser STT · Web Speech API implementation of the speech-to-text provider. · `src/voice/web-speech-stt.ts`
- [TTS stream](https://github.com/typhoidcoma/patyna/blob/main/src/audio/elevenlabs-tts.ts): text-to-speech · ElevenLabs streaming speech synthesis over WebSocket. · `src/audio/elevenlabs-tts.ts`
- [TTS player](https://github.com/typhoidcoma/patyna/blob/main/src/audio/tts-player.ts): audio playback · Buffers and feeds streaming speech into playback. · `src/audio/tts-player.ts`
- [Audio manager](https://github.com/typhoidcoma/patyna/blob/main/src/audio/audio-manager.ts): playback · Routes PCM and manages audio output lifecycle. · `src/audio/audio-manager.ts`
- [Scene manager](https://github.com/typhoidcoma/patyna/blob/main/src/scene/scene-manager.ts): rendering · Owns Three.js scene lifecycle, camera, and lighting. · `src/scene/scene-manager.ts`
- [Avatar controller](https://github.com/typhoidcoma/patyna/blob/main/src/scene/avatar-controller.ts): motion bridge · Applies tracking and state to avatar animation. · `src/scene/avatar-controller.ts`
- [Face tracking](https://github.com/typhoidcoma/patyna/blob/main/src/tracking/face-tracker.ts): vision · Webcam access and MediaPipe landmark tracking. · `src/tracking/face-tracker.ts`
- [Presence](https://github.com/typhoidcoma/patyna/blob/main/src/tracking/presence-manager.ts): state derivation · Derives user presence and visibility from tracking signals. · `src/tracking/presence-manager.ts`

### Backend integration

Realtime transport, REST APIs, and quest/task domain translation.

- [WebSocket client](https://github.com/typhoidcoma/patyna/blob/main/src/comm/websocket-client.ts): realtime transport · Reconnectable client for the backend message stream. · `src/comm/websocket-client.ts`
- [Message codec](https://github.com/typhoidcoma/patyna/blob/main/src/comm/message-codec.ts): serialization · Encodes and decodes protocol messages. · `src/comm/message-codec.ts`
- [Protocol](https://github.com/typhoidcoma/patyna/blob/main/src/comm/protocol.ts): message types · Shared message and event contract for realtime communication. · `src/comm/protocol.ts`
- [Aelora client](https://github.com/typhoidcoma/patyna/blob/main/src/api/aelora-client.ts): REST client · REST API access for users, sessions, memory, and mood. · `src/api/aelora-client.ts`
- [Quest logic](https://github.com/typhoidcoma/patyna/blob/main/src/quests/parse-add-task-intent.ts): domain translation · Intent parsing and quest-to-task mapping for task-oriented flows. · `src/quests/parse-add-task-intent.ts`

### Product shells

The separate demo and dashboard experiences built from the shared backend concepts.

- [P0 demo](https://github.com/typhoidcoma/patyna/blob/main/src/demo/demo-main.ts): demo shell · Lightweight dashboard experience with seeded state and widgets. · `src/demo/demo-main.ts`
- [LUMINORA](https://github.com/typhoidcoma/patyna/blob/main/src/demo2/demo2-main.ts): dashboard shell · Richer product shell with dashboard, vault, journal, and modals. · `src/demo2/demo2-main.ts`
- [Demo UI](https://github.com/typhoidcoma/patyna/blob/main/src/ui/demo-hud.ts): demo components · Reusable widgets and behavior for the P0 demo experience. · `src/ui/demo-hud.ts`
- [Dashboard UI](https://github.com/typhoidcoma/patyna/blob/main/src/demo2/components/modal-manager.ts): component set · Structured component set for the LUMINORA shell. · `src/demo2/components/modal-manager.ts`
- [Celebration FX](https://github.com/typhoidcoma/patyna/blob/main/src/fx/celebration.ts): effects · Lightweight celebration effects used by the demo experiences. · `src/fx/celebration.ts`

## Connections

- Main app → App orchestrator: boots · Primary entry initializes the application orchestrator.
- App orchestrator → Event bus: publishes · App-level events flow through the shared event bus.
- App orchestrator → State machine: drives · Lifecycle transitions are governed by the state machine.
- App orchestrator → Voice manager: wires · Orchestrator connects voice input to app behavior.
- App orchestrator → Scene manager: wires · Orchestrator connects the experience to rendering.
- Voice manager → VAD: uses · Voice manager consumes speech activity detection.
- Voice manager → STT provider: selects · Voice manager talks through the STT abstraction.
- STT provider → Web Speech STT: implements · Browser speech recognition is one STT provider implementation.
- Voice manager → TTS stream: streams · Voice output is synthesized through streaming TTS.
- TTS stream → TTS player: feeds · Speech audio is handed to the player for playback.
- TTS player → Audio manager: routes · Playback is managed by the shared audio subsystem.
- Audio manager → Scene manager: analyzes · Audio output amplitude can inform avatar animation sync.
- Face tracking → Presence: derives · Presence state is computed from camera and landmark signals.
- Face tracking → Avatar controller: feeds · Tracking data drives avatar motion and orientation.
- Presence → Avatar controller: modulates · Presence affects dimming and eyes-closed behavior.
- Scene manager → Avatar controller: hosts · Rendering manager owns avatar presentation lifecycle.
- WebSocket client → Message codec: encodes · Realtime transport uses the shared message codec.
- Message codec → Protocol: conforms to · Codec is defined by the protocol contract.
- Aelora client → Quest logic: supplies data · REST-backed content feeds quest/task translation.
- P0 demo → Demo UI: composes · P0 demo is built from reusable dashboard widgets.
- LUMINORA → Dashboard UI: composes · LUMINORA shell is built from its component set.
- P0 demo → Aelora client: syncs · Demo experience can consume backend data.
- LUMINORA → Aelora client: syncs · Dashboard shell consumes backend memory and session data.
- LUMINORA → Quest logic: maps tasks · Task-oriented UI relies on quest translation logic.
- LUMINORA → Event bus: listens · Dashboard interactions can use shared app events.
- P0 demo → Celebration FX: celebrates · Task completion may trigger celebratory effects.
