gildardojair/sentivision

Purpose and deliverables

AURA BUAP × IRIS UDLAP is a browser-only multimodal analysis system for research/service-experience sessions. It captures webcam and microphone streams locally, infers facial affect and vocal prosody, records session telemetry, and exports CSV. A separate standalone dashboard loads exported CSVs locally to produce an aggregate session report.

Primary entry points:

• Live capture app: index.html, orchestrated by app.js

• Offline session dashboard: analizar-sesion.html

• Styling/shared presentation: style.css

Live application orchestration

app.js is the central control plane: it initializes browser permissions and models, runs the real-time sampling loop, updates UI state, selects/comparisons emotion engines, maintains in-memory session records, detects key moments, and serializes CSV at download time. Reloading without export discards state.

The application requires a secure origin or localhost for getUserMedia; it has no backend. Initial model dependencies are downloaded from CDNs and subsequently browser-cached.

Grounding: index.html, app.js, style.css.

Vision and demographic pipeline

MediaPipe Face Landmarker is wrapped behind facemesh.js. It supplies face bounding boxes, 478 3D landmarks, a facial transformation matrix, and 52 blendshapes. This is the source boundary for face count/geometry, head roll-yaw-pitch, eye features, and emotion-model inputs.

Age/gender remains a face-api.js capability, fed a MediaPipe-derived face crop rather than its legacy detector. This separates current facial detection/tracking inputs from demographic inference.

Grounding: facemesh.js, app.js, index.html.

Emotion engines and affect derivation

Three emotion sources can be evaluated live: face-api baseline expressions, heuristic late fusion, and the default trained MLP. The MLP consumes 52 blendshapes, applies embedded network inference and temporal stabilization, and loads trained weights from a repository asset. The fusion engine combines visual blendshapes with voice-derived features.

Emotion probabilities and pose/eye signals are transformed into valence, arousal, Russell-style quadrants, nearest affect labels, positivity, and attention. These derived outputs are downstream of raw inference and are not clinical claims.

Grounding: emotion-ml.js, emotion_ml_weights.json, emotion-fusion.js; affect.js.

Audio, interaction, and satisfaction flow

voice.js uses Web Audio APIs to derive pitch, RMS energy, and vocal activation locally. A manual Client/Advisor role selection is an explicit boundary: the system does not perform speaker diarization.

When the selected speaker is Client, facial valence and vocal activation feed an EMA satisfaction signal. The orchestration layer flags positive/negative key moments and obtains a participant self-evaluation before final CSV generation.

Grounding: voice.js, app.js, analizar-sesion.html.

Quality, persistence, and reporting

Real-time quality metrics operate over sliding windows: detection coverage, expression entropy, and dominant-emotion flip rate. They inform live degradation alerts and are included in the session export.

The exported UTF-8 BOM CSV contains sampled observations, quality data, and key-moment sections. analizar-sesion.html is intentionally self-contained: it parses user-selected session CSV files in-browser and renders timeline, affect quadrants, key moments, and a verdict. Its verdict assumes a single person per recording; multi-person re-identification is not implemented.

Grounding: metrics.js, app.js, analizar-sesion.html.

Evaluation and historical boundary

Offline notebooks and archived browser experiments support benchmarking and prior implementation investigation but are outside the production execution path. The archive contains superseded app snapshots and isolated evaluation/diagnostic tooling; the root application modules define the active architecture.

Grounding: notebooks/evaluacion_edad_genero_3pipelines.ipynb, archive/app_v2.js, archive/evaluacion-offline/.