# rhy404/affect-aware-dynamic-ui architecture

> Architecture diagram of the GitHub repository rhy404/affect-aware-dynamic-ui, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rhy404/affect-aware-dynamic-ui
- Repository: https://github.com/rhy404/affect-aware-dynamic-ui

## Overview

Affect-Aware Dynamic UI detects cognitive load from meeting audio/video and adapts a Teams-style interface. The main path is webcam/microphone capture, live feature construction, model prediction and smoothing, then WebSocket state delivery into the React adaptation policy and meeting views. The repository also includes AVCAffe window preprocessing and multimodal model training, plus validation-window replay and manual state control. Evidence is limited to the supplied excerpts, SOURCE INDEX references, and README; unsampled model and UI internals are represented only at documented subsystem level.

Main entry points: [`frontend/src/App.js`](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/App.js) (UI state orchestration), [`backend/app.py`](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/backend/app.py) (FastAPI routes and state), [`preprocessing/audio_features.py`](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/preprocessing/audio_features.py) (Audio features), [`preprocessing/video_features.py`](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/preprocessing/video_features.py) (Video features)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_audio["Audio features<br/>[audio_features.py]"]
  node_video["Video features<br/>[video_features.py]"]
  node_windows["Window assembly<br/>[windows.py]"]
  node_clipreader["ClipReader dataset<br/>[dataset.py]"]
end

subgraph group_model["Model development"]
  node_training["Model training"]
end

subgraph group_backend["Inference backend"]
  node_validation["Validation replay"]
  node_livefeatures["Live tensor building<br/>[features.py]"]
  node_engine["Prediction and smoothing<br/>[inference.py]"]
  node_api["FastAPI routes and state<br/>[app.py]"]
end

subgraph group_frontend["Adaptive meeting UI"]
  node_capture["Live capture"]
  node_backendhook["Backend connection<br/>[useBackend.js]"]
  node_app["UI state orchestration<br/>[App.js]"]
  node_policy["Adaptation policy<br/>[config.js]"]
  node_meeting["Meeting simulation<br/>[useMeetingSim.js]"]
  node_demopanel["Demo controls<br/>[DemoPanel.js]"]
  node_video_grid["Adaptive video layout<br/>[VideoGrid.js]"]
  node_sidepanel["Chat and activity<br/>[SidePanel.js]"]
  node_toasts["Notification display<br/>[Toasts.js]"]
  node_topbar["Meeting controls<br/>[TopBar.js]"]
end

node_dataset[("AVCAffe clips")]
node_browser(("Meeting participant"))

node_dataset -->|"provides audio"| node_audio
node_dataset -->|"provides crops"| node_video
node_audio -->|"computes features"| node_windows
node_video -->|"computes features"| node_windows
node_windows -->|"supplies windows"| node_clipreader
node_clipreader -->|"feeds training"| node_training
node_training -->|"exports model"| node_engine
node_browser -->|"shares media"| node_capture
node_capture -->|"uploads live window"| node_api
node_audio -->|"reuses preprocessing"| node_livefeatures
node_video -->|"reuses preprocessing"| node_livefeatures
node_livefeatures -->|"builds model tensors"| node_engine
node_engine -->|"returns prediction"| node_api
node_validation -->|"posts replay window"| node_api
node_api -->|"pushes state"| node_backendhook
node_backendhook -->|"delivers state"| node_app
node_app -->|"selects adaptation"| node_policy
node_app -->|"starts sensing"| node_capture
node_app -->|"connects backend"| node_backendhook
node_app -->|"receives meeting events"| node_meeting
node_app -->|"renders controls"| node_demopanel
node_app -->|"renders layout"| node_video_grid
node_app -->|"renders panels"| node_sidepanel
node_app -->|"renders notifications"| node_toasts
node_app -->|"renders controls"| node_topbar
node_browser -->|"chooses manual state"| node_app

click node_audio "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/preprocessing/audio_features.py"
click node_video "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/preprocessing/video_features.py"
click node_windows "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/preprocessing/windows.py"
click node_clipreader "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/preprocessing/dataset.py"
click node_training "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/models/train_multimodal.py"
click node_validation "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/backend/replay_val_clips.py"
click node_capture "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/hooks/useLiveInference.js"
click node_livefeatures "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/backend/features.py"
click node_engine "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/backend/inference.py"
click node_api "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/backend/app.py"
click node_backendhook "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/hooks/useBackend.js"
click node_app "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/App.js"
click node_policy "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/config.js"
click node_meeting "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/hooks/useMeetingSim.js"
click node_demopanel "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/components/DemoPanel.js"
click node_video_grid "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/components/VideoGrid.js"
click node_sidepanel "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/components/SidePanel.js"
click node_toasts "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/components/Toasts.js"
click node_topbar "https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/components/TopBar.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_audio,node_video,node_windows,node_clipreader toneBlue
class node_training,node_dataset toneAmber
class node_validation,node_livefeatures,node_engine,node_api toneMint
class node_capture,node_backendhook,node_app,node_policy,node_meeting,node_demopanel,node_video_grid,node_sidepanel,node_toasts,node_topbar toneRose
class node_browser toneIndigo
```

## Components

### Data preparation

- [Audio features](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/preprocessing/audio_features.py): `preprocessing/audio_features.py`
- [Video features](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/preprocessing/video_features.py): `preprocessing/video_features.py`
- [Window assembly](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/preprocessing/windows.py): `preprocessing/windows.py`
- [ClipReader dataset](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/preprocessing/dataset.py): `preprocessing/dataset.py`

### Model development

- [Model training](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/models/train_multimodal.py): `models/train_multimodal.py`

### Inference backend

- [Validation replay](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/backend/replay_val_clips.py): `backend/replay_val_clips.py`
- [Live tensor building](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/backend/features.py): `backend/features.py`
- [Prediction and smoothing](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/backend/inference.py): `backend/inference.py`
- [FastAPI routes and state](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/backend/app.py): `backend/app.py`

### Adaptive meeting UI

- [Live capture](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/hooks/useLiveInference.js): `frontend/src/hooks/useLiveInference.js`
- [Backend connection](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/hooks/useBackend.js): `frontend/src/hooks/useBackend.js`
- [UI state orchestration](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/App.js): `frontend/src/App.js`
- [Adaptation policy](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/config.js): `frontend/src/config.js`
- [Meeting simulation](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/hooks/useMeetingSim.js): `frontend/src/hooks/useMeetingSim.js`
- [Demo controls](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/components/DemoPanel.js): `frontend/src/components/DemoPanel.js`
- [Adaptive video layout](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/components/VideoGrid.js): `frontend/src/components/VideoGrid.js`
- [Chat and activity](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/components/SidePanel.js): `frontend/src/components/SidePanel.js`
- [Notification display](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/components/Toasts.js): `frontend/src/components/Toasts.js`
- [Meeting controls](https://github.com/rhy404/affect-aware-dynamic-ui/blob/main/frontend/src/components/TopBar.js): `frontend/src/components/TopBar.js`

### Other components

- **AVCAffe clips**
- **Meeting participant**

## Connections

- AVCAffe clips → Audio features: provides audio
- AVCAffe clips → Video features: provides crops
- Audio features → Window assembly: computes features (evidence: `preprocessing/windows.py`)
- Video features → Window assembly: computes features (evidence: `preprocessing/windows.py`)
- Window assembly → ClipReader dataset: supplies windows (evidence: `preprocessing/dataset.py`)
- ClipReader dataset → Model training: feeds training
- Model training → Prediction and smoothing: exports model
- Meeting participant → Live capture: shares media (evidence: `frontend/src/hooks/useLiveInference.js`)
- Live capture → FastAPI routes and state: uploads live window
- Audio features → Live tensor building: reuses preprocessing
- Video features → Live tensor building: reuses preprocessing
- Live tensor building → Prediction and smoothing: builds model tensors (evidence: `backend/app.py`)
- Prediction and smoothing → FastAPI routes and state: returns prediction (evidence: `backend/app.py`)
- Validation replay → FastAPI routes and state: posts replay window (evidence: `backend/app.py`)
- FastAPI routes and state → Backend connection: pushes state
- Backend connection → UI state orchestration: delivers state (evidence: `frontend/src/App.js`)
- UI state orchestration → Adaptation policy: selects adaptation (evidence: `frontend/src/App.js`)
- UI state orchestration → Live capture: starts sensing (evidence: `frontend/src/App.js`)
- UI state orchestration → Backend connection: connects backend (evidence: `frontend/src/App.js`)
- UI state orchestration → Meeting simulation: receives meeting events (evidence: `frontend/src/App.js`)
- UI state orchestration → Demo controls: renders controls (evidence: `frontend/src/App.js`)
- UI state orchestration → Adaptive video layout: renders layout (evidence: `frontend/src/App.js`)
- UI state orchestration → Chat and activity: renders panels (evidence: `frontend/src/App.js`)
- UI state orchestration → Notification display: renders notifications (evidence: `frontend/src/App.js`)
- UI state orchestration → Meeting controls: renders controls (evidence: `frontend/src/App.js`)
- Meeting participant → UI state orchestration: chooses manual state
