# gildardojair/sentivision architecture

> Architecture diagram of the GitHub repository gildardojair/sentivision, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gildardojair/sentivision
- Repository: https://github.com/gildardojair/sentivision

## Overview

#### Purpose and deliverables
AURA BUAP × IRIS UDLAP is a browser-only multimodal analysis system for research/service-experience sessions. It captures webcam and microphone streams locally, infers facial affect and vocal prosody, records session telemetry, and exports CSV. A separate standalone dashboard loads exported CSVs locally to produce an aggregate session report.

Primary entry points:
- Live capture app: `index.html`, orchestrated by `app.js`
- Offline session dashboard: `analizar-sesion.html`
- Styling/shared presentation: `style.css`

#### Live application orchestration
`app.js` is the central control plane: it initializes browser permissions and models, runs the real-time sampling loop, updates UI state, selects/comparisons emotion engines, maintains in-memory session records, detects key moments, and serializes CSV at download time. Reloading without export discards state.

The application requires a secure origin or localhost for `getUserMedia`; it has no backend. Initial model dependencies are downloaded from CDNs and subsequently browser-cached.

Grounding: `index.html`, `app.js`, `style.css`.

#### Vision and demographic pipeline
MediaPipe Face Landmarker is wrapped behind `facemesh.js`. It supplies face bounding boxes, 478 3D landmarks, a facial transformation matrix, and 52 blendshapes. This is the source boundary for face count/geometry, head roll-yaw-pitch, eye features, and emotion-model inputs.

Age/gender remains a face-api.js capability, fed a MediaPipe-derived face crop rather than its legacy detector. This separates current facial detection/tracking inputs from demographic inference.

Grounding: `facemesh.js`, `app.js`, `index.html`.

#### Emotion engines and affect derivation
Three emotion sources can be evaluated live: face-api baseline expressions, heuristic late fusion, and the default trained MLP. The MLP consumes 52 blendshapes, applies embedded network inference and temporal stabilization, and loads trained weights from a repository asset. The fusion engine combines visual blendshapes with voice-derived features.

Emotion probabilities and pose/eye signals are transformed into valence, arousal, Russell-style quadrants, nearest affect labels, positivity, and attention. These derived outputs are downstream of raw inference and are not clinical claims.

Grounding: `emotion-ml.js`, `emotion_ml_weights.json`, `emotion-fusion.js`; `affect.js`.

#### Audio, interaction, and satisfaction flow
`voice.js` uses Web Audio APIs to derive pitch, RMS energy, and vocal activation locally. A manual Client/Advisor role selection is an explicit boundary: the system does not perform speaker diarization.

When the selected speaker is Client, facial valence and vocal activation feed an EMA satisfaction signal. The orchestration layer flags positive/negative key moments and obtains a participant self-evaluation before final CSV generation.

Grounding: `voice.js`, `app.js`, `analizar-sesion.html`.

#### Quality, persistence, and reporting
Real-time quality metrics operate over sliding windows: detection coverage, expression entropy, and dominant-emotion flip rate. They inform live degradation alerts and are included in the session export.

The exported UTF-8 BOM CSV contains sampled observations, quality data, and key-moment sections. `analizar-sesion.html` is intentionally self-contained: it parses user-selected session CSV files in-browser and renders timeline, affect quadrants, key moments, and a verdict. Its verdict assumes a single person per recording; multi-person re-identification is not implemented.

Grounding: `metrics.js`, `app.js`, `analizar-sesion.html`.

#### Evaluation and historical boundary
Offline notebooks and archived browser experiments support benchmarking and prior implementation investigation but are outside the production execution path. The archive contains superseded app snapshots and isolated evaluation/diagnostic tooling; the root application modules define the active architecture.

Grounding: `notebooks/evaluacion_edad_genero_3pipelines.ipynb`, `archive/app_v2.js`, `archive/evaluacion-offline/`.

Main entry points: [`app.js`](https://github.com/gildardojair/sentivision/blob/main/app.js) (Session orchestrator), [`facemesh.js`](https://github.com/gildardojair/sentivision/blob/main/facemesh.js) (Face Landmarker wrapper), [`affect.js`](https://github.com/gildardojair/sentivision/blob/main/affect.js) (Affect derivation), [`voice.js`](https://github.com/gildardojair/sentivision/blob/main/voice.js) (Voice features)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_live["Live browser app"]
  node_capture_ui["Live capture UI<br/>browser entry<br/>[index.html]"]
  node_orchestrator["Session orchestrator<br/>control plane<br/>[app.js]"]
  node_presentation["Shared presentation<br/>styles<br/>[style.css]"]
  node_browser_media(("Webcam and microphone<br/>browser media APIs"))
  node_role_self_eval["Role and self-evaluation<br/>manual interaction<br/>[app.js]"]
  node_quality_metrics["Quality metrics<br/>sliding-window metrics<br/>[metrics.js]"]
  node_session_records[("In-memory session records<br/>ephemeral state<br/>[app.js]")]
end

subgraph group_inference["Inference pipeline"]
  node_face_landmarker["Face Landmarker wrapper<br/>vision source<br/>[facemesh.js]"]
  node_demographics["Age and gender inference<br/>face-api capability<br/>[app.js]"]
  node_voice_analysis["Voice features<br/>Web Audio analysis<br/>[voice.js]"]
  node_mlp_emotion["Trained MLP emotion<br/>default emotion engine<br/>[emotion-ml.js]"]
  node_mlp_weights["MLP weights<br/>model asset"]
  node_fusion_emotion["Heuristic fusion emotion<br/>emotion engine<br/>[emotion-fusion.js]"]
  node_faceapi_emotion["Face-api expressions<br/>baseline emotion engine<br/>[app.js]"]
  node_affect_derivation["Affect derivation<br/>derived signals<br/>[affect.js]"]
end

subgraph group_reporting["Export and reporting"]
  node_csv_export["Session CSV export<br/>UTF-8 BOM CSV<br/>[app.js]"]
  node_dashboard["Offline session dashboard<br/>browser entry"]
end

subgraph group_historical["Evaluation history"]
  node_offline_evaluation["Offline evaluation notebook<br/>benchmark artifact"]
  node_archived_app["Archived app snapshot<br/>superseded implementation<br/>[app_v2.js]"]
  node_archive_tools["Archived evaluation tools<br/>diagnostic tooling"]
end

node_capture_ui -->|"starts"| node_orchestrator
node_presentation -->|"styles"| node_capture_ui
node_browser_media -->|"media streams"| node_orchestrator
node_orchestrator -->|"video frames"| node_face_landmarker
node_face_landmarker -->|"face crop"| node_demographics
node_orchestrator -->|"audio stream"| node_voice_analysis
node_face_landmarker -->|"52 blendshapes"| node_mlp_emotion
node_mlp_weights -->|"loads"| node_mlp_emotion
node_face_landmarker -->|"visual features"| node_fusion_emotion
node_voice_analysis -->|"vocal features"| node_fusion_emotion
node_face_landmarker -->|"face input"| node_faceapi_emotion
node_mlp_emotion -->|"emotion probabilities"| node_affect_derivation
node_fusion_emotion -->|"emotion probabilities"| node_affect_derivation
node_faceapi_emotion -.->|"baseline probabilities"| node_affect_derivation
node_role_self_eval -->|"selected role and rating"| node_orchestrator
node_affect_derivation -->|"affect and satisfaction inputs"| node_orchestrator
node_voice_analysis -->|"vocal activation"| node_orchestrator
node_face_landmarker -->|"detection signals"| node_quality_metrics
node_affect_derivation -->|"emotion stream"| node_quality_metrics
node_orchestrator -->|"sampled observations and moments"| node_session_records
node_quality_metrics -->|"quality windows"| node_session_records
node_session_records -->|"serialize on download"| node_csv_export
node_csv_export -->|"user-selected local file"| node_dashboard

click node_capture_ui "https://github.com/gildardojair/sentivision/blob/main/index.html"
click node_orchestrator "https://github.com/gildardojair/sentivision/blob/main/app.js"
click node_presentation "https://github.com/gildardojair/sentivision/blob/main/style.css"
click node_role_self_eval "https://github.com/gildardojair/sentivision/blob/main/app.js"
click node_face_landmarker "https://github.com/gildardojair/sentivision/blob/main/facemesh.js"
click node_demographics "https://github.com/gildardojair/sentivision/blob/main/app.js"
click node_voice_analysis "https://github.com/gildardojair/sentivision/blob/main/voice.js"
click node_mlp_emotion "https://github.com/gildardojair/sentivision/blob/main/emotion-ml.js"
click node_mlp_weights "https://github.com/gildardojair/sentivision/blob/main/emotion_ml_weights.json"
click node_fusion_emotion "https://github.com/gildardojair/sentivision/blob/main/emotion-fusion.js"
click node_faceapi_emotion "https://github.com/gildardojair/sentivision/blob/main/app.js"
click node_affect_derivation "https://github.com/gildardojair/sentivision/blob/main/affect.js"
click node_quality_metrics "https://github.com/gildardojair/sentivision/blob/main/metrics.js"
click node_session_records "https://github.com/gildardojair/sentivision/blob/main/app.js"
click node_csv_export "https://github.com/gildardojair/sentivision/blob/main/app.js"
click node_dashboard "https://github.com/gildardojair/sentivision/blob/main/analizar-sesion.html"
click node_offline_evaluation "https://github.com/gildardojair/sentivision/blob/main/notebooks/evaluacion_edad_genero_3pipelines.ipynb"
click node_archived_app "https://github.com/gildardojair/sentivision/blob/main/archive/app_v2.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_capture_ui,node_orchestrator,node_presentation,node_browser_media,node_role_self_eval,node_quality_metrics,node_session_records toneBlue
class node_face_landmarker,node_demographics,node_voice_analysis,node_mlp_emotion,node_mlp_weights,node_fusion_emotion,node_faceapi_emotion,node_affect_derivation toneAmber
class node_csv_export,node_dashboard toneMint
class node_offline_evaluation,node_archived_app,node_archive_tools toneRose
```

## Components

### Live browser app

Local capture, orchestration, and session state.

- [Live capture UI](https://github.com/gildardojair/sentivision/blob/main/index.html): browser entry · `index.html`
- [Session orchestrator](https://github.com/gildardojair/sentivision/blob/main/app.js): control plane · Owns permissions, sampling, UI state, records, and CSV serialization. · `app.js`
- [Shared presentation](https://github.com/gildardojair/sentivision/blob/main/style.css): styles · `style.css`
- **Webcam and microphone**: browser media APIs · Requires secure origin or localhost.
- [Role and self-evaluation](https://github.com/gildardojair/sentivision/blob/main/app.js): manual interaction · Client/Advisor selection replaces speaker diarization. · `app.js`
- [Quality metrics](https://github.com/gildardojair/sentivision/blob/main/metrics.js): sliding-window metrics · Tracks coverage, entropy, and emotion flip rate. · `metrics.js`
- [In-memory session records](https://github.com/gildardojair/sentivision/blob/main/app.js): ephemeral state · Reloading without export discards state. · `app.js`

### Inference pipeline

On-device multimodal analysis.

- [Face Landmarker wrapper](https://github.com/gildardojair/sentivision/blob/main/facemesh.js): vision source · Provides face geometry, pose, landmarks, and blendshapes. · `facemesh.js`
- [Age and gender inference](https://github.com/gildardojair/sentivision/blob/main/app.js): face-api capability · Uses a MediaPipe-derived face crop. · `app.js`
- [Voice features](https://github.com/gildardojair/sentivision/blob/main/voice.js): Web Audio analysis · Derives pitch, RMS energy, and vocal activation locally. · `voice.js`
- [Trained MLP emotion](https://github.com/gildardojair/sentivision/blob/main/emotion-ml.js): default emotion engine · Runs embedded inference and temporal stabilization over 52 blendshapes. · `emotion-ml.js`
- [MLP weights](https://github.com/gildardojair/sentivision/blob/main/emotion_ml_weights.json): model asset · `emotion_ml_weights.json`
- [Heuristic fusion emotion](https://github.com/gildardojair/sentivision/blob/main/emotion-fusion.js): emotion engine · Combines visual blendshapes and voice features. · `emotion-fusion.js`
- [Face-api expressions](https://github.com/gildardojair/sentivision/blob/main/app.js): baseline emotion engine · `app.js`
- [Affect derivation](https://github.com/gildardojair/sentivision/blob/main/affect.js): derived signals · Produces valence, arousal, quadrants, labels, positivity, and attention. · `affect.js`

### Export and reporting

Local session export and offline review.

- [Session CSV export](https://github.com/gildardojair/sentivision/blob/main/app.js): UTF-8 BOM CSV · Contains observations, quality data, and key moments. · `app.js`
- [Offline session dashboard](https://github.com/gildardojair/sentivision/blob/main/analizar-sesion.html): browser entry · Locally parses CSVs and renders the aggregate report. · `analizar-sesion.html`

### Evaluation history

Outside the production execution path.

- [Offline evaluation notebook](https://github.com/gildardojair/sentivision/blob/main/notebooks/evaluacion_edad_genero_3pipelines.ipynb): benchmark artifact · Outside the production path. · `notebooks/evaluacion_edad_genero_3pipelines.ipynb`
- [Archived app snapshot](https://github.com/gildardojair/sentivision/blob/main/archive/app_v2.js): superseded implementation · Outside the production path. · `archive/app_v2.js`
- **Archived evaluation tools**: diagnostic tooling · Outside the production path.

## Connections

- Live capture UI → Session orchestrator: starts
- Shared presentation → Live capture UI: styles
- Webcam and microphone → Session orchestrator: media streams
- Session orchestrator → Face Landmarker wrapper: video frames
- Face Landmarker wrapper → Age and gender inference: face crop
- Session orchestrator → Voice features: audio stream
- Face Landmarker wrapper → Trained MLP emotion: 52 blendshapes
- MLP weights → Trained MLP emotion: loads
- Face Landmarker wrapper → Heuristic fusion emotion: visual features
- Voice features → Heuristic fusion emotion: vocal features
- Face Landmarker wrapper → Face-api expressions: face input
- Trained MLP emotion → Affect derivation: emotion probabilities
- Heuristic fusion emotion → Affect derivation: emotion probabilities
- Face-api expressions → Affect derivation: baseline probabilities
- Role and self-evaluation → Session orchestrator: selected role and rating
- Affect derivation → Session orchestrator: affect and satisfaction inputs
- Voice features → Session orchestrator: vocal activation
- Face Landmarker wrapper → Quality metrics: detection signals
- Affect derivation → Quality metrics: emotion stream
- Session orchestrator → In-memory session records: sampled observations and moments
- Quality metrics → In-memory session records: quality windows
- In-memory session records → Session CSV export: serialize on download
- Session CSV export → Offline session dashboard: user-selected local file