sophiamyang/finger-frame-effect-lucy

Purpose and entry points

This is a static browser demo for a “finger frame” effect: a user frames an area with both hands, and the framed region displays a live AI video-to-video transformation. The app is served directly as static files, with index.html as the browser entry point and main.js containing the application behavior. examples/lucy.mp4 is a demonstration asset, not part of the live pipeline.

Browser UI and rendering surface

index.html defines the full-screen interactive page, including controls for API-key entry, style selection, and custom prompts. main.js owns webcam setup, UI event handling, and composition of camera and transformed video onto a canvas. The camera view is mirrored; the AI output is clipped/mapped to the detected finger-frame quadrilateral.

Paths: index.html, main.js

Camera acquisition and local fallback

The primary source is a 1280×720, 30fps webcam MediaStream, requested from the browser after user permission. That stream supplies both local rendering and the remote AI session. If no Decart key/session is available, the framed region falls back to a local color-filter effect so hand tracking remains usable. A ?demo mode substitutes a synthetic feed/fake hands and disables AI.

Paths: main.js, index.html

Hand tracking and frame geometry

MediaPipe Hand Landmarker runs client-side against the camera feed. Its output is converted into an ordered four-corner “finger frame” quad. The tracking stage includes gesture qualification and temporal stabilization described by the project: anatomical corner ordering, hysteresis, teleport rejection, velocity-adaptive smoothing, and temporary dropout holding. This is the key boundary between raw hand landmarks and rendering geometry.

Paths: main.js

Realtime AI transport

With a user-provided Decart platform key, the camera stream is sent to Decart Lucy 2.5 through a WebRTC session. The service returns a transformed live video track; the browser draws this returned track only inside the tracked quad while retaining the original mirrored camera outside it. API-key storage and use remain browser-side, but the key is necessarily passed to Decart for session establishment.

Paths: main.js, index.html

Prompt/style control flow

Keyboard shortcuts 1–6 and UI controls select predefined visual styles. A custom style uses the prompt entered in the key/prompt panel. Prompt changes update the active Lucy session rather than reconnecting, separating session lifecycle from style mutation.

Paths: main.js, index.html

Runtime and deployment boundary

There is no backend in this repository. The runtime is a modern browser with camera permission, Canvas/video APIs, MediaPipe hand tracking, WebRTC, and the external Decart realtime model service. Development/deployment is static hosting; the documented local server is sufficient because browser camera access requires an HTTP(S) origin.

Paths: README.md, index.html, main.js