# compsmart/talking-face architecture

> Architecture diagram of the GitHub repository compsmart/talking-face, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/compsmart/talking-face
- Repository: https://github.com/compsmart/talking-face

## Overview

FACE CLONER is a browser-based talking 3D face: users connect to Gemini Live, chat by text, and can map a portrait onto the avatar and generate a matching persona. The main workflow runs from UI input through the app bootstrap into rendering, conversation, and synchronized audio-driven animation. The README documents several renderer and audio details whose implementations were not sampled; those links reflect documented behavior, while uncertain wiring is omitted. The mobile UI shares the runtime. The optional server proxy is shown separately.

Main entry points: [`src/main.js`](https://github.com/compsmart/talking-face/blob/main/src/main.js) (App bootstrap), [`src/gemini/live-client.js`](https://github.com/compsmart/talking-face/blob/main/src/gemini/live-client.js) (Gemini Live client), [`src/gpu/renderer.js`](https://github.com/compsmart/talking-face/blob/main/src/gpu/renderer.js) (WebGPU renderer), [`src/ui/hud.js`](https://github.com/compsmart/talking-face/blob/main/src/ui/hud.js) (Desktop HUD)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["User interface"]
  node_desktop["Desktop shell<br/>[index.html]"]
  node_hud["Desktop HUD<br/>[hud.js]"]
  node_mobile["Mobile UI<br/>[app.js]"]
  node_mobilehud["Touch HUD<br/>[mobile-hud.js]"]
  node_main["App bootstrap<br/>[main.js]"]
end

subgraph group_avatar["Face and rendering"]
  node_mesh["Canonical mesh<br/>[mesh.js]"]
  node_morphs["Morph rig<br/>[morphs.js]"]
  node_shape["Face adaptation<br/>[shape.js]"]
  node_mapper["Photo mapping<br/>[mapper.js]"]
  node_renderer["WebGPU renderer<br/>[renderer.js]"]
  node_shaders["GPU shaders<br/>[shaders.js]"]
end

subgraph group_conversation["Gemini conversation"]
  node_live["Gemini Live client<br/>[live-client.js]"]
  node_tools["Face tools<br/>[face-tools.js]"]
  node_persona["Persona generation<br/>[persona.js]"]
end

subgraph group_audio["Speech animation"]
  node_animator["Expression animator<br/>[animator.js]"]
  node_lipsync["Predictive lip sync<br/>[lipsync.js]"]
  node_player["PCM player<br/>[player.js]"]
end

subgraph group_proxy["Optional proxy"]
  node_proxy["Token and WS proxy<br/>[server.js]"]
end

node_user(("User"))
node_gemini{{"Google Gemini API"}}
node_speaker(("Audio output"))

node_user -->|"uses"| node_hud
node_user -->|"uses"| node_mobilehud
node_desktop -->|"hosts"| node_hud
node_mobile -->|"extends"| node_mobilehud
node_hud -->|"sends input"| node_main
node_mobile -->|"bootstraps"| node_main
node_main -->|"loads"| node_mesh
node_main -->|"builds rig"| node_morphs
node_main -->|"creates"| node_renderer
node_main -->|"maps photo"| node_mapper
node_main -->|"adapts mesh"| node_shape
node_main -->|"requests persona"| node_persona
node_main -->|"connects"| node_live
node_main -->|"updates"| node_animator
node_main -->|"runs tool calls"| node_tools
node_mapper -->|"returns UVs and landmarks"| node_main
node_shape -->|"returns adapted geometry"| node_main
node_main -->|"sets photo and geometry"| node_renderer
node_renderer -->|"executes"| node_shaders
node_live -->|"sends session; receives audio and calls"| node_gemini
node_live -->|"delivers events"| node_main
node_main -->|"sends tool responses"| node_live
node_persona -->|"requests photo analysis"| node_gemini
node_main -->|"feeds audio chunks"| node_lipsync
node_main -->|"schedules PCM"| node_player
node_lipsync -->|"provides mouth weights"| node_animator
node_animator -->|"provides morph weights"| node_renderer
node_player -->|"plays audio"| node_speaker
node_gemini -.->|"proxied Live traffic"| node_proxy
node_live -.->|"requests token and WS"| node_proxy
node_main -.->|"optional persona scan"| node_proxy

click node_desktop "https://github.com/compsmart/talking-face/blob/main/index.html"
click node_hud "https://github.com/compsmart/talking-face/blob/main/src/ui/hud.js"
click node_mobile "https://github.com/compsmart/talking-face/blob/main/mobile/app.js"
click node_mobilehud "https://github.com/compsmart/talking-face/blob/main/mobile/mobile-hud.js"
click node_main "https://github.com/compsmart/talking-face/blob/main/src/main.js"
click node_mesh "https://github.com/compsmart/talking-face/blob/main/src/face/mesh.js"
click node_morphs "https://github.com/compsmart/talking-face/blob/main/src/face/morphs.js"
click node_shape "https://github.com/compsmart/talking-face/blob/main/src/photo/shape.js"
click node_mapper "https://github.com/compsmart/talking-face/blob/main/src/photo/mapper.js"
click node_renderer "https://github.com/compsmart/talking-face/blob/main/src/gpu/renderer.js"
click node_shaders "https://github.com/compsmart/talking-face/blob/main/src/gpu/shaders.js"
click node_live "https://github.com/compsmart/talking-face/blob/main/src/gemini/live-client.js"
click node_tools "https://github.com/compsmart/talking-face/blob/main/src/gemini/face-tools.js"
click node_persona "https://github.com/compsmart/talking-face/blob/main/src/gemini/persona.js"
click node_animator "https://github.com/compsmart/talking-face/blob/main/src/face/animator.js"
click node_lipsync "https://github.com/compsmart/talking-face/blob/main/src/audio/lipsync.js"
click node_player "https://github.com/compsmart/talking-face/blob/main/src/audio/player.js"
click node_proxy "https://github.com/compsmart/talking-face/blob/main/server/server.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_desktop,node_hud,node_mobile,node_mobilehud,node_main,node_user toneBlue
class node_mesh,node_morphs,node_shape,node_mapper,node_renderer,node_shaders toneAmber
class node_live,node_tools,node_persona,node_gemini toneMint
class node_animator,node_lipsync,node_player toneRose
class node_proxy,node_speaker toneIndigo
```

## Components

### User interface

- [Desktop shell](https://github.com/compsmart/talking-face/blob/main/index.html): `index.html`
- [Desktop HUD](https://github.com/compsmart/talking-face/blob/main/src/ui/hud.js): `src/ui/hud.js`
- [Mobile UI](https://github.com/compsmart/talking-face/blob/main/mobile/app.js): `mobile/app.js`
- [Touch HUD](https://github.com/compsmart/talking-face/blob/main/mobile/mobile-hud.js): `mobile/mobile-hud.js`
- [App bootstrap](https://github.com/compsmart/talking-face/blob/main/src/main.js): `src/main.js`

### Face and rendering

- [Canonical mesh](https://github.com/compsmart/talking-face/blob/main/src/face/mesh.js): `src/face/mesh.js`
- [Morph rig](https://github.com/compsmart/talking-face/blob/main/src/face/morphs.js): `src/face/morphs.js`
- [Face adaptation](https://github.com/compsmart/talking-face/blob/main/src/photo/shape.js): `src/photo/shape.js`
- [Photo mapping](https://github.com/compsmart/talking-face/blob/main/src/photo/mapper.js): `src/photo/mapper.js`
- [WebGPU renderer](https://github.com/compsmart/talking-face/blob/main/src/gpu/renderer.js): `src/gpu/renderer.js`
- [GPU shaders](https://github.com/compsmart/talking-face/blob/main/src/gpu/shaders.js): `src/gpu/shaders.js`

### Gemini conversation

- [Gemini Live client](https://github.com/compsmart/talking-face/blob/main/src/gemini/live-client.js): `src/gemini/live-client.js`
- [Face tools](https://github.com/compsmart/talking-face/blob/main/src/gemini/face-tools.js): `src/gemini/face-tools.js`
- [Persona generation](https://github.com/compsmart/talking-face/blob/main/src/gemini/persona.js): `src/gemini/persona.js`

### Speech animation

- [Expression animator](https://github.com/compsmart/talking-face/blob/main/src/face/animator.js): `src/face/animator.js`
- [Predictive lip sync](https://github.com/compsmart/talking-face/blob/main/src/audio/lipsync.js): `src/audio/lipsync.js`
- [PCM player](https://github.com/compsmart/talking-face/blob/main/src/audio/player.js): `src/audio/player.js`

### Optional proxy

- [Token and WS proxy](https://github.com/compsmart/talking-face/blob/main/server/server.js): `server/server.js`

### Other components

- **User**
- **Google Gemini API**
- **Audio output**

## Connections

- User → Desktop HUD: uses
- User → Touch HUD: uses
- Desktop shell → Desktop HUD: hosts
- Mobile UI → Touch HUD: extends
- Desktop HUD → App bootstrap: sends input
- Mobile UI → App bootstrap: bootstraps
- App bootstrap → Canonical mesh: loads
- App bootstrap → Morph rig: builds rig
- App bootstrap → WebGPU renderer: creates
- App bootstrap → Photo mapping: maps photo
- App bootstrap → Face adaptation: adapts mesh
- App bootstrap → Persona generation: requests persona
- App bootstrap → Gemini Live client: connects
- App bootstrap → Expression animator: updates
- App bootstrap → Face tools: runs tool calls
- Photo mapping → App bootstrap: returns UVs and landmarks
- Face adaptation → App bootstrap: returns adapted geometry
- App bootstrap → WebGPU renderer: sets photo and geometry
- WebGPU renderer → GPU shaders: executes
- Gemini Live client → Google Gemini API: sends session; receives audio and calls
- Gemini Live client → App bootstrap: delivers events
- App bootstrap → Gemini Live client: sends tool responses
- Persona generation → Google Gemini API: requests photo analysis
- App bootstrap → Predictive lip sync: feeds audio chunks
- App bootstrap → PCM player: schedules PCM
- Predictive lip sync → Expression animator: provides mouth weights
- Expression animator → WebGPU renderer: provides morph weights
- PCM player → Audio output: plays audio
- Google Gemini API → Token and WS proxy: proxied Live traffic
- Gemini Live client → Token and WS proxy: requests token and WS
- App bootstrap → Token and WS proxy: optional persona scan