How ericwei97-cloud/gesture-synth works
Gesture Synth is a browser-based musical instrument: camera input feeds hand tracking, whose gestures are stabilized and translated into chord and expression controls for Web Audio synthesis and visual feedback. The sampled source places these responsibilities in one file, so nodes distinguish runtime stages rather than separate modules. The README documents the left-hand chord and right-hand voicing/expression roles; excerpts do not show enough detail to map each gesture to a specific musical parameter.
Main entry points: main.js Musical State
Components 12
Camera and Tracking
- Camera Capture
main.js - Hand Tracking
main.js - Hand Landmarks
main.js
Gesture to Music
- Gesture Analysis
main.js - Gesture Stabilizers
main.js - Musical State
main.js - Chord and Key Mapping
main.js - Expression Mapping
main.js
Sound and Interface
- Synth Engine
main.js - Audio Output
- Visual Feedback
main.js
Other components
- Player
Connections 12
- Player → Camera Capture: provides video
main.js - Camera Capture → Hand Tracking: supplies frames
main.js - Hand Tracking → Hand Landmarks: detects hands
main.js - Hand Landmarks → Gesture Analysis: analyzes landmarks
main.js - Gesture Analysis → Gesture Stabilizers: stabilizes gestures
main.js - Gesture Stabilizers → Musical State: produces state
main.js - Musical State → Chord and Key Mapping: selects chords
main.js - Musical State → Expression Mapping: maps expression
main.js - Expression Mapping → Synth Engine: sets expression
main.js - Synth Engine → Audio Output: outputs audio
main.js - Chord and Key Mapping → Synth Engine: sets notes
main.js - Musical State → Visual Feedback: updates display
main.js