How typhoidcoma/patyna works
1. Overall shape
This repo is a browser-based, multi-entry application built with Vite + TypeScript. It ships three user-facing experiences from the same codebase: the main real-time avatar app (/), a text-only P0 demo (/demo.html), and the more polished LUMINORA dashboard (/demo2.html). The architecture is centered on a shared realtime “avatar + conversation” core, with two separate dashboard/product shells layered on top.
2. Entry points and orchestration
src/main.ts and src/app.ts are the main app bootstrap/orchestrator path. The two demos have their own bootstraps and app/state modules under src/demo/ and src/demo2/. This separation is important: the realtime avatar engine is shared conceptually, but each experience owns its own UI layout, state model, and user flow.
3. Core runtime model
src/core/ provides the app-wide coordination primitives: a typed event bus and a finite state machine that drives the interaction states (idle, listening, thinking, speaking). This core layer sits above media, scene, and backend code and acts as the glue between user input, network responses, and avatar animation.
4. Real-time voice pipeline
The voice stack is split across src/voice/ and src/audio/. Voice input uses voice activity detection plus a speech-to-text provider abstraction, with a Web Speech API implementation in web-speech-stt.ts. Voice output uses ElevenLabs streaming TTS over WebSocket, then routes PCM into an AudioWorklet-backed playback path via tts-player.ts and audio-manager.ts. The audio output is also analyzed for amplitude so the avatar can animate in sync with speech.
5. 3D avatar and visual scene
src/scene/ contains the Three.js rendering layer: scene setup, avatar mesh, environment/background, mood-driven animation profiles, and color mapping. avatar-controller.ts bridges tracking data into avatar motion, while scene-manager.ts owns rendering lifecycle and camera/lights. This layer is purely presentation-focused and consumes higher-level state such as speaking/mood/presence.
6. Camera, face tracking, and presence
src/tracking/ handles webcam access, MediaPipe face landmarks, and presence detection. Presence is derived from face visibility over time and feeds the avatar’s dimming/eyes-closed behavior. This is a distinct subsystem boundary: camera input is not just for rendering, but also for presence semantics that affect the whole experience.
7. Backend communication and memory
src/comm/ contains the WebSocket protocol, message codec, and reconnecting client used to talk to the Aelora backend. src/api/aelora-client.ts handles REST calls for users, sessions, memory, and mood. In practice, the backend is the app’s external brain: chat responses, user-scoped memory/vault data, and session state live there, while the browser maintains the interactive front end.
8. Domain-specific quest/task logic
src/quests/ is the app’s domain translation layer for task-oriented interactions. It parses intents from free text, maps quests to tasks, subscribes/fetches quest data, and matches quests by title hints. This is the logic that turns conversational input or backend content into actionable dashboard items in the demo experiences.
9. Product shells: P0 demo and LUMINORA
src/demo/ is a lightweight dashboard experience with seeded state, drag-and-drop widgets, and celebration effects for task completion. src/demo2/ is the richer LUMINORA product shell: it has its own componentized dashboard, modal system, journal input, vault/memory view, weekly rhythm UI, and task/goal panels. These shells consume the shared backend/voice concepts but do not depend on the full 3D camera experience.
10. UI, styling, and interaction plumbing
src/ui/ contains reusable HUD/sidebar widgets and drag behavior for floating panels; src/demo2/components/ is a more structured component set for the LUMINORA dashboard. Global styling is split between shared styles in src/styles/ and experience-specific CSS in src/demo2/demo2.css and src/demo/demo-overrides.css. This is a mostly client-side UI architecture with no separate frontend framework layer.
11. External services and operational boundaries
The app depends on ElevenLabs for streaming TTS, Aelora for chat/memory/session APIs, and optionally Supabase-related plumbing for auth or feedback (src/auth/, src/api/feedback-client.ts). The browser is the primary runtime, with a worker in public/workers/ supporting audio playback. Static assets and HTML entrypoints are served by Vite, making this a multi-page browser application rather than a single SPA bundle.
12. How data flows through the system
Typical flow is: user input via mic/camera or text UI enters the core event/state system, which coordinates voice recognition, backend calls, and avatar state. Backend responses and TTS stream back into the audio subsystem, while the scene layer reacts to speaking amplitude, mood, and presence. In the dashboard modes, task and memory data flow from Aelora/local demo state into UI panels, with completion events triggering celebratory effects and modal updates.
Main entry points: app.ts App orchestrator · demo2-main.ts LUMINORA · voice-manager.ts Voice manager · demo-main.ts P0 demo
Components 25
Browser runtime
Shared bootstrap, app coordination, and cross-cutting state.
- Main app · Primary browser entry for the real-time avatar experience.
src/main.ts - App orchestrator · Top-level application wiring and lifecycle coordination.
src/app.ts - Event bus · Typed pub/sub channel for app-wide events.
src/core/event-bus.ts - State machine · Interaction state control for idle, listening, thinking, and speaking.
src/core/state-machine.ts
Avatar experience
Voice, tracking, audio, and 3D presentation for the real-time app.
- Voice manager · Coordinates speech input, VAD, and provider selection.
src/voice/voice-manager.ts - VAD · Detects active speech segments from microphone input.
src/voice/vad.ts - STT provider · Abstract speech recognition interface used by the voice stack.
src/voice/stt-provider.ts - Web Speech STT · Web Speech API implementation of the speech-to-text provider.
src/voice/web-speech-stt.ts - TTS stream · ElevenLabs streaming speech synthesis over WebSocket.
src/audio/elevenlabs-tts.ts - TTS player · Buffers and feeds streaming speech into playback.
src/audio/tts-player.ts - Audio manager · Routes PCM and manages audio output lifecycle.
src/audio/audio-manager.ts - Scene manager · Owns Three.js scene lifecycle, camera, and lighting.
src/scene/scene-manager.ts - Avatar controller · Applies tracking and state to avatar animation.
src/scene/avatar-controller.ts - Face tracking · Webcam access and MediaPipe landmark tracking.
src/tracking/face-tracker.ts - Presence · Derives user presence and visibility from tracking signals.
src/tracking/presence-manager.ts
Backend integration
Realtime transport, REST APIs, and quest/task domain translation.
- WebSocket client · Reconnectable client for the backend message stream.
src/comm/websocket-client.ts - Message codec · Encodes and decodes protocol messages.
src/comm/message-codec.ts - Protocol · Shared message and event contract for realtime communication.
src/comm/protocol.ts - Aelora client · REST API access for users, sessions, memory, and mood.
src/api/aelora-client.ts - Quest logic · Intent parsing and quest-to-task mapping for task-oriented flows.
src/quests/parse-add-task-intent.ts
Product shells
The separate demo and dashboard experiences built from the shared backend concepts.
- P0 demo · Lightweight dashboard experience with seeded state and widgets.
src/demo/demo-main.ts - LUMINORA · Richer product shell with dashboard, vault, journal, and modals.
src/demo2/demo2-main.ts - Demo UI · Reusable widgets and behavior for the P0 demo experience.
src/ui/demo-hud.ts - Dashboard UI · Structured component set for the LUMINORA shell.
src/demo2/components/modal-manager.ts - Celebration FX · Lightweight celebration effects used by the demo experiences.
src/fx/celebration.ts
Connections 26
- Main app → App orchestrator: boots · Primary entry initializes the application orchestrator.
- App orchestrator → Event bus: publishes · App-level events flow through the shared event bus.
- App orchestrator → State machine: drives · Lifecycle transitions are governed by the state machine.
- App orchestrator → Voice manager: wires · Orchestrator connects voice input to app behavior.
- App orchestrator → Scene manager: wires · Orchestrator connects the experience to rendering.
- Voice manager → VAD: uses · Voice manager consumes speech activity detection.
- Voice manager → STT provider: selects · Voice manager talks through the STT abstraction.
- STT provider → Web Speech STT: implements · Browser speech recognition is one STT provider implementation.
- Voice manager → TTS stream: streams · Voice output is synthesized through streaming TTS.
- TTS stream → TTS player: feeds · Speech audio is handed to the player for playback.
- TTS player → Audio manager: routes · Playback is managed by the shared audio subsystem.
- Audio manager → Scene manager: analyzes · Audio output amplitude can inform avatar animation sync.
- Face tracking → Presence: derives · Presence state is computed from camera and landmark signals.
- Face tracking → Avatar controller: feeds · Tracking data drives avatar motion and orientation.
- Presence → Avatar controller: modulates · Presence affects dimming and eyes-closed behavior.
- Scene manager → Avatar controller: hosts · Rendering manager owns avatar presentation lifecycle.
- WebSocket client → Message codec: encodes · Realtime transport uses the shared message codec.
- Message codec → Protocol: conforms to · Codec is defined by the protocol contract.
- Aelora client → Quest logic: supplies data · REST-backed content feeds quest/task translation.
- P0 demo → Demo UI: composes · P0 demo is built from reusable dashboard widgets.
- LUMINORA → Dashboard UI: composes · LUMINORA shell is built from its component set.
- P0 demo → Aelora client: syncs · Demo experience can consume backend data.
- LUMINORA → Aelora client: syncs · Dashboard shell consumes backend memory and session data.
- LUMINORA → Quest logic: maps tasks · Task-oriented UI relies on quest translation logic.
- LUMINORA → Event bus: listens · Dashboard interactions can use shared app events.
- P0 demo → Celebration FX: celebrates · Task completion may trigger celebratory effects.