How fusical/emotiw works
Fusical is a research pipeline for three-way group sentiment prediction from short videos. Its documented workflow preprocesses video into scene frames, facial data, pose keypoints, and audio features, then applies modality-specific classifiers and fuses their predictions. Image-captioning and YOLO-based detection are additional documented-by-code capabilities, though their role in the final ensemble is not established here. Source excerpts show the preprocessing orchestrator and several classifier-to-generator calls; ensemble wiring is documented in the README, while some modality handoffs are not directly shown.
Main entry points: preprocess_all_modes.py Preprocess workflow · ensemble Prediction ensemble · frames_classifier.py Scene classifier · pose_classifier.py Pose classifier
Components 18
Preprocessing
- Preprocess workflow
src/preprocessors/preprocess_all_modes.py - Scene frames
src/preprocessors/scene_preprocessor.py - Face extraction
src/preprocessors/face_preprocessor.py - Facial expression features
src/preprocessors/fer_preprocessor.py - Pose keypoints
src/preprocessors/pose_preprocessor.py - Audio embeddings
src/preprocessors/audio_preprocessor.py
Modality models
- Frame batches
src/generators/frame_generator.py - Pose batches
src/generators/pose_generator.py - Scene classifier
src/classifiers/frames_classifier.py - Pose classifier
src/classifiers/pose_classifier.py - Face classifier
src/classifiers/face_classifier.py - Audio classifier
src/classifiers/audio_classifier.py - Caption classifier
src/classifiers/image_captioning_classifier.py - YOLO detector
src/classifiers/yolo3_one_file_to_detect_them_all.py
Fusion and extensions
- Prediction ensemble
notebooks/ensemble - Sentiment classes
Other components
- Researcher
- Input videos
Connections 20
- Researcher → Preprocess workflow: runs preprocessing
src/preprocessors/preprocess_all_modes.py - Input videos → Preprocess workflow: supplies videos
src/preprocessors/preprocess_all_modes.py - Preprocess workflow → Scene frames: runs
src/preprocessors/preprocess_all_modes.py - Preprocess workflow → Face extraction: runs
src/preprocessors/preprocess_all_modes.py - Preprocess workflow → Pose keypoints: runs
src/preprocessors/preprocess_all_modes.py - Preprocess workflow → Audio embeddings: runs
src/preprocessors/preprocess_all_modes.py - Preprocess workflow → Facial expression features: configures
src/preprocessors/preprocess_all_modes.py - Scene classifier → Frame batches: uses batches
src/classifiers/frames_classifier.py - Pose classifier → Pose batches: uses batches
src/classifiers/pose_classifier.py - Scene frames → Scene classifier: provides modality data
- Pose keypoints → Pose classifier: provides modality data
- Facial expression features → Face classifier: provides modality data
- Audio embeddings → Audio classifier: provides embeddings
- Scene classifier → Prediction ensemble: contributes predictions
README.md - Pose classifier → Prediction ensemble: contributes predictions
README.md - Face classifier → Prediction ensemble: contributes predictions
README.md - Audio classifier → Prediction ensemble: contributes predictions
README.md - Prediction ensemble → Sentiment classes: predicts classes
README.md - Caption classifier → Sentiment classes: classifies sentiment
- YOLO detector → Scene frames: detects objects in