# fusical/emotiw architecture

> Architecture diagram of the GitHub repository fusical/emotiw, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fusical/emotiw
- Repository: https://github.com/fusical/emotiw

## Overview

Fusical is a research pipeline for three-way group sentiment prediction from short videos. Its documented workflow preprocesses video into scene frames, facial data, pose keypoints, and audio features, then applies modality-specific classifiers and fuses their predictions. Image-captioning and YOLO-based detection are additional documented-by-code capabilities, though their role in the final ensemble is not established here. Source excerpts show the preprocessing orchestrator and several classifier-to-generator calls; ensemble wiring is documented in the README, while some modality handoffs are not directly shown.

Main entry points: [`src/preprocessors/preprocess_all_modes.py`](https://github.com/fusical/emotiw/blob/master/src/preprocessors/preprocess_all_modes.py) (Preprocess workflow), [`notebooks/ensemble`](https://github.com/fusical/emotiw/tree/master/notebooks/ensemble) (Prediction ensemble), [`src/classifiers/frames_classifier.py`](https://github.com/fusical/emotiw/blob/master/src/classifiers/frames_classifier.py) (Scene classifier), [`src/classifiers/pose_classifier.py`](https://github.com/fusical/emotiw/blob/master/src/classifiers/pose_classifier.py) (Pose classifier)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Preprocessing"]
  node_preprocess["Preprocess workflow"]
  node_scene["Scene frames"]
  node_face["Face extraction"]
  node_fer["Facial expression features"]
  node_pose["Pose keypoints"]
  node_audio["Audio embeddings"]
end

subgraph group_models["Modality models"]
  node_framegen["Frame batches<br/>[frame_generator.py]"]
  node_posegen["Pose batches<br/>[pose_generator.py]"]
  node_frameclf["Scene classifier"]
  node_poseclf["Pose classifier<br/>[pose_classifier.py]"]
  node_faceclf["Face classifier<br/>[face_classifier.py]"]
  node_audioclf["Audio classifier"]
  node_captionclf["Caption classifier"]
  node_yolo["YOLO detector"]
end

subgraph group_fusion["Fusion and extensions"]
  node_ensemble["Prediction ensemble"]
  node_sentiment["Sentiment classes"]
end

node_researcher(("Researcher"))
node_videos["Input videos"]

node_researcher -->|"runs preprocessing"| node_preprocess
node_videos -->|"supplies videos"| node_preprocess
node_preprocess -->|"runs"| node_scene
node_preprocess -->|"runs"| node_face
node_preprocess -->|"runs"| node_pose
node_preprocess -->|"runs"| node_audio
node_preprocess -->|"configures"| node_fer
node_frameclf -->|"uses batches"| node_framegen
node_poseclf -->|"uses batches"| node_posegen
node_scene -.->|"provides modality data"| node_frameclf
node_pose -.->|"provides modality data"| node_poseclf
node_fer -.->|"provides modality data"| node_faceclf
node_audio -.->|"provides embeddings"| node_audioclf
node_frameclf -->|"contributes predictions"| node_ensemble
node_poseclf -->|"contributes predictions"| node_ensemble
node_faceclf -->|"contributes predictions"| node_ensemble
node_audioclf -->|"contributes predictions"| node_ensemble
node_ensemble -->|"predicts classes"| node_sentiment
node_captionclf -.->|"classifies sentiment"| node_sentiment
node_yolo -.->|"detects objects in"| node_scene

click node_preprocess "https://github.com/fusical/emotiw/blob/master/src/preprocessors/preprocess_all_modes.py"
click node_scene "https://github.com/fusical/emotiw/blob/master/src/preprocessors/scene_preprocessor.py"
click node_face "https://github.com/fusical/emotiw/blob/master/src/preprocessors/face_preprocessor.py"
click node_fer "https://github.com/fusical/emotiw/blob/master/src/preprocessors/fer_preprocessor.py"
click node_pose "https://github.com/fusical/emotiw/blob/master/src/preprocessors/pose_preprocessor.py"
click node_audio "https://github.com/fusical/emotiw/blob/master/src/preprocessors/audio_preprocessor.py"
click node_framegen "https://github.com/fusical/emotiw/blob/master/src/generators/frame_generator.py"
click node_posegen "https://github.com/fusical/emotiw/blob/master/src/generators/pose_generator.py"
click node_frameclf "https://github.com/fusical/emotiw/blob/master/src/classifiers/frames_classifier.py"
click node_poseclf "https://github.com/fusical/emotiw/blob/master/src/classifiers/pose_classifier.py"
click node_faceclf "https://github.com/fusical/emotiw/blob/master/src/classifiers/face_classifier.py"
click node_audioclf "https://github.com/fusical/emotiw/blob/master/src/classifiers/audio_classifier.py"
click node_captionclf "https://github.com/fusical/emotiw/blob/master/src/classifiers/image_captioning_classifier.py"
click node_yolo "https://github.com/fusical/emotiw/blob/master/src/classifiers/yolo3_one_file_to_detect_them_all.py"
click node_ensemble "https://github.com/fusical/emotiw/tree/master/notebooks/ensemble"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_preprocess,node_scene,node_face,node_fer,node_pose,node_audio toneBlue
class node_framegen,node_posegen,node_frameclf,node_poseclf,node_faceclf,node_audioclf,node_captionclf,node_yolo toneAmber
class node_ensemble,node_sentiment toneMint
class node_researcher,node_videos toneIndigo
```

## Components

### Preprocessing

- [Preprocess workflow](https://github.com/fusical/emotiw/blob/master/src/preprocessors/preprocess_all_modes.py): `src/preprocessors/preprocess_all_modes.py`
- [Scene frames](https://github.com/fusical/emotiw/blob/master/src/preprocessors/scene_preprocessor.py): `src/preprocessors/scene_preprocessor.py`
- [Face extraction](https://github.com/fusical/emotiw/blob/master/src/preprocessors/face_preprocessor.py): `src/preprocessors/face_preprocessor.py`
- [Facial expression features](https://github.com/fusical/emotiw/blob/master/src/preprocessors/fer_preprocessor.py): `src/preprocessors/fer_preprocessor.py`
- [Pose keypoints](https://github.com/fusical/emotiw/blob/master/src/preprocessors/pose_preprocessor.py): `src/preprocessors/pose_preprocessor.py`
- [Audio embeddings](https://github.com/fusical/emotiw/blob/master/src/preprocessors/audio_preprocessor.py): `src/preprocessors/audio_preprocessor.py`

### Modality models

- [Frame batches](https://github.com/fusical/emotiw/blob/master/src/generators/frame_generator.py): `src/generators/frame_generator.py`
- [Pose batches](https://github.com/fusical/emotiw/blob/master/src/generators/pose_generator.py): `src/generators/pose_generator.py`
- [Scene classifier](https://github.com/fusical/emotiw/blob/master/src/classifiers/frames_classifier.py): `src/classifiers/frames_classifier.py`
- [Pose classifier](https://github.com/fusical/emotiw/blob/master/src/classifiers/pose_classifier.py): `src/classifiers/pose_classifier.py`
- [Face classifier](https://github.com/fusical/emotiw/blob/master/src/classifiers/face_classifier.py): `src/classifiers/face_classifier.py`
- [Audio classifier](https://github.com/fusical/emotiw/blob/master/src/classifiers/audio_classifier.py): `src/classifiers/audio_classifier.py`
- [Caption classifier](https://github.com/fusical/emotiw/blob/master/src/classifiers/image_captioning_classifier.py): `src/classifiers/image_captioning_classifier.py`
- [YOLO detector](https://github.com/fusical/emotiw/blob/master/src/classifiers/yolo3_one_file_to_detect_them_all.py): `src/classifiers/yolo3_one_file_to_detect_them_all.py`

### Fusion and extensions

- [Prediction ensemble](https://github.com/fusical/emotiw/tree/master/notebooks/ensemble): `notebooks/ensemble`
- **Sentiment classes**

### Other components

- **Researcher**
- **Input videos**

## Connections

- Researcher → Preprocess workflow: runs preprocessing (evidence: `src/preprocessors/preprocess_all_modes.py`)
- Input videos → Preprocess workflow: supplies videos (evidence: `src/preprocessors/preprocess_all_modes.py`)
- Preprocess workflow → Scene frames: runs (evidence: `src/preprocessors/preprocess_all_modes.py`)
- Preprocess workflow → Face extraction: runs (evidence: `src/preprocessors/preprocess_all_modes.py`)
- Preprocess workflow → Pose keypoints: runs (evidence: `src/preprocessors/preprocess_all_modes.py`)
- Preprocess workflow → Audio embeddings: runs (evidence: `src/preprocessors/preprocess_all_modes.py`)
- Preprocess workflow → Facial expression features: configures (evidence: `src/preprocessors/preprocess_all_modes.py`)
- Scene classifier → Frame batches: uses batches (evidence: `src/classifiers/frames_classifier.py`)
- Pose classifier → Pose batches: uses batches (evidence: `src/classifiers/pose_classifier.py`)
- Scene frames → Scene classifier: provides modality data
- Pose keypoints → Pose classifier: provides modality data
- Facial expression features → Face classifier: provides modality data
- Audio embeddings → Audio classifier: provides embeddings
- Scene classifier → Prediction ensemble: contributes predictions (evidence: `README.md`)
- Pose classifier → Prediction ensemble: contributes predictions (evidence: `README.md`)
- Face classifier → Prediction ensemble: contributes predictions (evidence: `README.md`)
- Audio classifier → Prediction ensemble: contributes predictions (evidence: `README.md`)
- Prediction ensemble → Sentiment classes: predicts classes (evidence: `README.md`)
- Caption classifier → Sentiment classes: classifies sentiment
- YOLO detector → Scene frames: detects objects in
