How rhy404/affect-aware-dynamic-ui works
Affect-Aware Dynamic UI detects cognitive load from meeting audio/video and adapts a Teams-style interface. The main path is webcam/microphone capture, live feature construction, model prediction and smoothing, then WebSocket state delivery into the React adaptation policy and meeting views. The repository also includes AVCAffe window preprocessing and multimodal model training, plus validation-window replay and manual state control. Evidence is limited to the supplied excerpts, SOURCE INDEX references, and README; unsampled model and UI internals are represented only at documented subsystem level.
Main entry points: App.js UI state orchestration · app.py FastAPI routes and state · audio_features.py Audio features · video_features.py Video features
Components 21
Data preparation
- Audio features
preprocessing/audio_features.py - Video features
preprocessing/video_features.py - Window assembly
preprocessing/windows.py - ClipReader dataset
preprocessing/dataset.py
Model development
- Model training
models/train_multimodal.py
Inference backend
- Validation replay
backend/replay_val_clips.py - Live tensor building
backend/features.py - Prediction and smoothing
backend/inference.py - FastAPI routes and state
backend/app.py
Adaptive meeting UI
- Live capture
frontend/src/hooks/useLiveInference.js - Backend connection
frontend/src/hooks/useBackend.js - UI state orchestration
frontend/src/App.js - Adaptation policy
frontend/src/config.js - Meeting simulation
frontend/src/hooks/useMeetingSim.js - Demo controls
frontend/src/components/DemoPanel.js - Adaptive video layout
frontend/src/components/VideoGrid.js - Chat and activity
frontend/src/components/SidePanel.js - Notification display
frontend/src/components/Toasts.js - Meeting controls
frontend/src/components/TopBar.js
Other components
- AVCAffe clips
- Meeting participant
Connections 26
- AVCAffe clips → Audio features: provides audio
- AVCAffe clips → Video features: provides crops
- Audio features → Window assembly: computes features
preprocessing/windows.py - Video features → Window assembly: computes features
preprocessing/windows.py - Window assembly → ClipReader dataset: supplies windows
preprocessing/dataset.py - ClipReader dataset → Model training: feeds training
- Model training → Prediction and smoothing: exports model
- Meeting participant → Live capture: shares media
frontend/src/hooks/useLiveInference.js - Live capture → FastAPI routes and state: uploads live window
- Audio features → Live tensor building: reuses preprocessing
- Video features → Live tensor building: reuses preprocessing
- Live tensor building → Prediction and smoothing: builds model tensors
backend/app.py - Prediction and smoothing → FastAPI routes and state: returns prediction
backend/app.py - Validation replay → FastAPI routes and state: posts replay window
backend/app.py - FastAPI routes and state → Backend connection: pushes state
- Backend connection → UI state orchestration: delivers state
frontend/src/App.js - UI state orchestration → Adaptation policy: selects adaptation
frontend/src/App.js - UI state orchestration → Live capture: starts sensing
frontend/src/App.js - UI state orchestration → Backend connection: connects backend
frontend/src/App.js - UI state orchestration → Meeting simulation: receives meeting events
frontend/src/App.js - UI state orchestration → Demo controls: renders controls
frontend/src/App.js - UI state orchestration → Adaptive video layout: renders layout
frontend/src/App.js - UI state orchestration → Chat and activity: renders panels
frontend/src/App.js - UI state orchestration → Notification display: renders notifications
frontend/src/App.js - UI state orchestration → Meeting controls: renders controls
frontend/src/App.js - Meeting participant → UI state orchestration: chooses manual state