# met4citizen/talkinghead architecture

> Architecture diagram of the GitHub repository met4citizen/talkinghead, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/met4citizen/talkinghead
- Repository: https://github.com/met4citizen/talkinghead

## Overview

##### Project type and purpose
- This is primarily a front-end JavaScript library and demo application (“TalkingHead”) for real-time 3D avatars that speak and lip-sync in the browser.
- It provides a core ES-module (`modules/talkinghead.mjs`) plus auxiliary modules for lip-sync, dynamic bones, and audio playback. It loads 3D models (GLB), animations (FBX), audio assets and integrates with external TTS/AI services via HTTP/WebSocket proxies.

##### File structure
Top-level components:
- index.html + siteconfig.js – the test/demo web app UI and configuration
- modules/ – ES modules: core TalkingHead class, lipsync-xx, dynamicbones, playback worklet
- examples/ – standalone HTML demos showing different integrations (Azure, MP3, minimal)
- avatars/, animations/, poses/, audio/, images/, fonts/ – static assets (models, FBX, audio, textures, fonts)
- README.md – detailed project overview, API reference, usage patterns
- siteconfig.js – demo app resource URLs, API proxy endpoints

##### Insights from README
- Core features: real-time TTS + word-level timestamps for lip-sync in multiple languages, dynamic bones physics, external TTS/A.I. service integrations
- Architecture: module-based, dynamic imports via importmap, uses ThreeJS/WebGL for rendering, Web Audio API + AudioWorklet for low-latency audio
- Extensibility: pluggable lipsync modules, dynamic bones config, custom poses/gestures, streaming interface

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Browser Client-side
    subgraph "Browser (Client-side)"
        direction TB
        UI["UI Layer<br/>(index.html)"]:::frontend
        Config["Configuration<br/>(siteconfig.js)"]:::frontend
        Loader["ES Module Loader<br/>(importmap)"]:::frontend
        Core["TalkingHead Core Module"]:::frontend
        LS_EN["Lip-sync Module<br/>(English)"]:::frontend
        LS_FI["Lip-sync Module<br/>(Finnish)"]:::frontend
        LS_LT["Lip-sync Module<br/>(Lithuanian)"]:::frontend
        DynBones["Dynamic Bones Module"]:::frontend
        Worklet["Audio Playback Worklet"]:::frontend
        Renderer["ThreeJS Renderer & Scene Graph"]:::frontend
        AudioAPI["Web Audio API<br/>& GainNodes"]:::frontend
        AW["AudioWorklet"]:::frontend
        AL["Asset Loader<br/>(GLTFLoader/FBXLoader)"]:::frontend

        UI -->|"loads"| Loader
        Loader -->|"imports"| Core
        Core -->|"uses"| LS_EN
        Core -->|"uses"| LS_FI
        Core -->|"uses"| LS_LT
        Core -->|"uses"| DynBones
        Core -->|"initiates"| Worklet
        Core -->|"invokes"| Renderer
        Core -->|"requests assets"| AL
        Core -->|"plays audio via"| AudioAPI
        Worklet -->|"processes audio"| AW
        AW --> AudioAPI
        AudioAPI --> Renderer
    end

    %% Static Assets
    subgraph "Static Assets"
        direction TB
        Avatars[("avatars/")]:::asset
        Animations[("animations/")]:::asset
        Poses[("poses/")]:::asset
        AudioAssets[("audio/")]:::asset
        Images[("images/")]:::asset
        Fonts[("fonts/")]:::asset

        AL --> Avatars
        AL --> Animations
        AL --> Poses
        AL --> AudioAssets
        AL --> Images
        AL --> Fonts
    end

    %% Proxy Layer
    subgraph "Proxy Layer" 
        direction TB
        Proxy["TTS/AI Proxy Endpoints"]:::proxy
    end

    Core -->|"Fetch/WS → Audio+Timestamps"| Proxy

    %% External Services
    subgraph "External Services"
        direction TB
        Google["Google Cloud TTS"]:::external
        Azure["Microsoft Azure Speech SDK"]:::external
        Eleven["ElevenLabs TTS"]:::external
        OpenAI["OpenAI / Vertex AI / Grok"]:::external
        Whisper["whisper-web / llama.cpp"]:::external

        Proxy --> Google
        Proxy --> Azure
        Proxy --> Eleven
        Proxy --> OpenAI
        Proxy --> Whisper
        Google -->|Audio+Timestamps| Core
        Azure -->|Audio+Timestamps| Core
        Eleven -->|Audio+Timestamps| Core
        OpenAI -->|Audio+Timestamps| Core
        Whisper -->|Audio+Timestamps| Core
    end

    %% Offline Asset Pipeline
    subgraph "Offline Asset Pipeline"
        direction TB
        Blender["Blender Scripts<br/>(.py)"]:::frontend
    end

    Blender -->|"prepares assets"| Avatars
    Blender --> Avatars
    Blender --> Animations

    %% Demo Examples & Docs
    subgraph "Demo & Docs"
        direction TB
        AzureAudio["Azure Audio Streaming Demo"]:::frontend
        AzureBlend["Azure Blendshapes Demo"]:::frontend
        Minimal["Minimal Integration Demo"]:::frontend
        MP3["MP3 Playback Demo"]:::frontend
        Kokoro["Kokoro TTS Demo (.mjs)"]:::frontend
        README["Project Overview & API Reference"]:::frontend
        Gitattributes[".gitattributes"]:::frontend
        License["LICENSE"]:::frontend

        AzureAudio --> UI
        AzureBlend --> UI
        Minimal --> UI
        MP3 --> UI
        Kokoro --> UI
    end

    %% Click Events
    click UI "https://github.com/met4citizen/talkinghead/blob/main/index.html"
    click Config "https://github.com/met4citizen/talkinghead/blob/main/siteconfig.js"
    click Core "https://github.com/met4citizen/talkinghead/blob/main/modules/talkinghead.mjs"
    click LS_EN "https://github.com/met4citizen/talkinghead/blob/main/modules/lipsync-en.mjs"
    click LS_FI "https://github.com/met4citizen/talkinghead/blob/main/modules/lipsync-fi.mjs"
    click LS_LT "https://github.com/met4citizen/talkinghead/blob/main/modules/lipsync-lt.mjs"
    click DynBones "https://github.com/met4citizen/talkinghead/blob/main/modules/dynamicbones.mjs"
    click Worklet "https://github.com/met4citizen/talkinghead/blob/main/modules/playback-worklet.js"
    click Avatars "https://github.com/met4citizen/talkinghead/tree/main/avatars/"
    click Animations "https://github.com/met4citizen/talkinghead/tree/main/animations/"
    click Poses "https://github.com/met4citizen/talkinghead/tree/main/poses/"
    click AudioAssets "https://github.com/met4citizen/talkinghead/tree/main/audio/"
    click Images "https://github.com/met4citizen/talkinghead/tree/main/images/"
    click Fonts "https://github.com/met4citizen/talkinghead/tree/main/fonts/"
    click Blender "https://github.com/met4citizen/talkinghead/tree/main/blender/"
    click AzureAudio "https://github.com/met4citizen/talkinghead/blob/main/examples/azure-audio-streaming.html"
    click AzureBlend "https://github.com/met4citizen/talkinghead/blob/main/examples/azure-blendshapes.html"
    click Minimal "https://github.com/met4citizen/talkinghead/blob/main/examples/minimal.html"
    click MP3 "https://github.com/met4citizen/talkinghead/blob/main/examples/mp3.html"
    click Kokoro "https://github.com/met4citizen/talkinghead/blob/main/examples/kokorotts.mjs"
    click README "https://github.com/met4citizen/talkinghead/blob/main/README.md"
    click Gitattributes "https://github.com/met4citizen/talkinghead/blob/main/.gitattributes"
    click License "https://github.com/met4citizen/talkinghead/tree/main/LICENSE"

    %% Styles
    classDef frontend fill:#D0E8FF,stroke:#0366d6,stroke-width:1px
    classDef asset fill:#E2F5D6,stroke:#228B22,stroke-width:1px
    classDef proxy fill:#FFE8C0,stroke:#FF8C00,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#666666,stroke-dasharray: 2 2,stroke-width:1px
```
