How xeaver/emotionclip works
EmotionCLIP learns video-and-language emotion representations from aligned visual frames, captions, and sentiment signals, then evaluates learned features with dataset-specific classifiers. The principal training path is configured in main.py, which builds the YouTube dataset and model and hands training to the engine. Preprocessing scripts cover video discovery, acquisition, annotation, and dataset preparation. The graph uses documented workflow and source excerpts; preprocessing-script internals and some main.py wiring were not sampled, so uncertain links are omitted.
Main entry points: main.py Training entry · linear_eval.py Linear evaluation · base.py EmotionCLIP model · trainer.py Training engine
Components 22
Data preparation
- Find playlists
preprocessing/search_youtube.py - Fetch videos
preprocessing/download_videos.py - Prepare frames
preprocessing/frames_to_hdf5.py - Detect people
preprocessing/yolo_additions/detect_human.py - Sentiment logits
preprocessing/get_sentiment_logits.py - Neutral scores
preprocessing/get_neutral_scores.py - Regroup transcripts
preprocessing/regroup_transcripts.py - Build index
preprocessing/create_index.py
Datasets
- YouTube dataset
src/datasets/youtube_v3.py - Benchmark datasets
src/datasets
Representation learning
- Experiment config
src/options.py - Training entry
main.py - EmotionCLIP model
src/models/base.py - Training engine
src/engine/trainer.py - Contrastive loss
src/models/loss.py
Evaluation
- Evaluation engine
src/engine/evaluation.py - Linear evaluation
linear_eval.py - Linear classifier
linear_eval.py
Other components
- Researcher
- YouTube
- Prepared data
- Pretrained checkpoint
Connections 26
- Researcher → Find playlists: requests data
- YouTube → Find playlists: provides listings
- YouTube → Fetch videos: provides media
- Find playlists → Prepared data: writes metadata
- Fetch videos → Prepared data: writes video and transcripts
- Prepare frames → Prepared data: writes frame data
- Detect people → Prepared data: writes human boxes
- Sentiment logits → Prepared data: writes logits
- Neutral scores → Prepared data: writes scores
- Regroup transcripts → Prepared data: writes transcripts
- Build index → Prepared data: writes index
- Researcher → Training entry: starts training
- Training entry → Experiment config: configures
- Training entry → YouTube dataset: constructs dataset
- Prepared data → YouTube dataset: read by
- Training entry → EmotionCLIP model: creates
- Pretrained checkpoint → EmotionCLIP model: initializes weights
- Training entry → Training engine: starts training
- Training engine → EmotionCLIP model: runs forward pass
- Training engine → Contrastive loss: computes objective
- Training entry → Evaluation engine: configures evaluation
- Researcher → Linear evaluation: requests evaluation
- Linear evaluation → Benchmark datasets: constructs datasets
- Linear evaluation → EmotionCLIP model: extracts features
- Linear evaluation → Linear classifier: fits and scores
- Benchmark datasets → Prepared data: reads benchmark data