How compsmart/talking-face works
FACE CLONER is a browser-based talking 3D face: users connect to Gemini Live, chat by text, and can map a portrait onto the avatar and generate a matching persona. The main workflow runs from UI input through the app bootstrap into rendering, conversation, and synchronized audio-driven animation. The README documents several renderer and audio details whose implementations were not sampled; those links reflect documented behavior, while uncertain wiring is omitted. The mobile UI shares the runtime. The optional server proxy is shown separately.
Main entry points: main.js App bootstrap · live-client.js Gemini Live client · renderer.js WebGPU renderer · hud.js Desktop HUD
Components 21
User interface
- Desktop shell
index.html - Desktop HUD
src/ui/hud.js - Mobile UI
mobile/app.js - Touch HUD
mobile/mobile-hud.js - App bootstrap
src/main.js
Face and rendering
- Canonical mesh
src/face/mesh.js - Morph rig
src/face/morphs.js - Face adaptation
src/photo/shape.js - Photo mapping
src/photo/mapper.js - WebGPU renderer
src/gpu/renderer.js - GPU shaders
src/gpu/shaders.js
Gemini conversation
- Gemini Live client
src/gemini/live-client.js - Face tools
src/gemini/face-tools.js - Persona generation
src/gemini/persona.js
Speech animation
- Expression animator
src/face/animator.js - Predictive lip sync
src/audio/lipsync.js - PCM player
src/audio/player.js
Optional proxy
- Token and WS proxy
server/server.js
Other components
- User
- Google Gemini API
- Audio output
Connections 31
- User → Desktop HUD: uses
- User → Touch HUD: uses
- Desktop shell → Desktop HUD: hosts
- Mobile UI → Touch HUD: extends
- Desktop HUD → App bootstrap: sends input
- Mobile UI → App bootstrap: bootstraps
- App bootstrap → Canonical mesh: loads
- App bootstrap → Morph rig: builds rig
- App bootstrap → WebGPU renderer: creates
- App bootstrap → Photo mapping: maps photo
- App bootstrap → Face adaptation: adapts mesh
- App bootstrap → Persona generation: requests persona
- App bootstrap → Gemini Live client: connects
- App bootstrap → Expression animator: updates
- App bootstrap → Face tools: runs tool calls
- Photo mapping → App bootstrap: returns UVs and landmarks
- Face adaptation → App bootstrap: returns adapted geometry
- App bootstrap → WebGPU renderer: sets photo and geometry
- WebGPU renderer → GPU shaders: executes
- Gemini Live client → Google Gemini API: sends session; receives audio and calls
- Gemini Live client → App bootstrap: delivers events
- App bootstrap → Gemini Live client: sends tool responses
- Persona generation → Google Gemini API: requests photo analysis
- App bootstrap → Predictive lip sync: feeds audio chunks
- App bootstrap → PCM player: schedules PCM
- Predictive lip sync → Expression animator: provides mouth weights
- Expression animator → WebGPU renderer: provides morph weights
- PCM player → Audio output: plays audio
- Google Gemini API → Token and WS proxy: proxied Live traffic
- Gemini Live client → Token and WS proxy: requests token and WS
- App bootstrap → Token and WS proxy: optional persona scan