# xeaver/emotionclip architecture

> Architecture diagram of the GitHub repository xeaver/emotionclip, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xeaver/emotionclip
- Repository: https://github.com/xeaver/emotionclip

## Overview

EmotionCLIP learns video-and-language emotion representations from aligned visual frames, captions, and sentiment signals, then evaluates learned features with dataset-specific classifiers. The principal training path is configured in main.py, which builds the YouTube dataset and model and hands training to the engine. Preprocessing scripts cover video discovery, acquisition, annotation, and dataset preparation. The graph uses documented workflow and source excerpts; preprocessing-script internals and some main.py wiring were not sampled, so uncertain links are omitted.

Main entry points: [`main.py`](https://github.com/xeaver/emotionclip/blob/main/main.py) (Training entry), [`linear_eval.py`](https://github.com/xeaver/emotionclip/blob/main/linear_eval.py) (Linear evaluation), [`src/models/base.py`](https://github.com/xeaver/emotionclip/blob/main/src/models/base.py) (EmotionCLIP model), [`src/engine/trainer.py`](https://github.com/xeaver/emotionclip/blob/main/src/engine/trainer.py) (Training engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Data preparation"]
  node_search["Find playlists<br/>[search_youtube.py]"]
  node_download["Fetch videos<br/>[download_videos.py]"]
  node_frames["Prepare frames<br/>[frames_to_hdf5.py]"]
  node_detect["Detect people<br/>[detect_human.py]"]
  node_sentiment["Sentiment logits"]
  node_neutral["Neutral scores"]
  node_regroup["Regroup transcripts"]
  node_index["Build index<br/>[create_index.py]"]
end

subgraph group_data["Datasets"]
  node_youtube_data["YouTube dataset<br/>[youtube_v3.py]"]
  node_other_data["Benchmark datasets"]
end

subgraph group_learn["Representation learning"]
  node_config["Experiment config<br/>[options.py]"]
  node_entry["Training entry<br/>[main.py]"]
  node_model["EmotionCLIP model<br/>[base.py]"]
  node_trainer["Training engine<br/>[trainer.py]"]
  node_loss["Contrastive loss<br/>[loss.py]"]
end

subgraph group_eval["Evaluation"]
  node_evaluation["Evaluation engine<br/>[evaluation.py]"]
  node_linear["Linear evaluation<br/>[linear_eval.py]"]
  node_classifier["Linear classifier<br/>[linear_eval.py]"]
end

node_researcher(("Researcher"))
node_youtube(("YouTube"))
node_dataset_files["Prepared data"]
node_checkpoint["Pretrained checkpoint"]

node_researcher -->|"requests data"| node_search
node_youtube -->|"provides listings"| node_search
node_youtube -->|"provides media"| node_download
node_search -->|"writes metadata"| node_dataset_files
node_download -->|"writes video and transcripts"| node_dataset_files
node_frames -->|"writes frame data"| node_dataset_files
node_detect -->|"writes human boxes"| node_dataset_files
node_sentiment -->|"writes logits"| node_dataset_files
node_neutral -->|"writes scores"| node_dataset_files
node_regroup -->|"writes transcripts"| node_dataset_files
node_index -->|"writes index"| node_dataset_files
node_researcher -->|"starts training"| node_entry
node_entry -->|"configures"| node_config
node_entry -->|"constructs dataset"| node_youtube_data
node_dataset_files -->|"read by"| node_youtube_data
node_entry -->|"creates"| node_model
node_checkpoint -.->|"initializes weights"| node_model
node_entry -->|"starts training"| node_trainer
node_trainer -->|"runs forward pass"| node_model
node_trainer -->|"computes objective"| node_loss
node_entry -->|"configures evaluation"| node_evaluation
node_researcher -->|"requests evaluation"| node_linear
node_linear -->|"constructs datasets"| node_other_data
node_linear -->|"extracts features"| node_model
node_linear -->|"fits and scores"| node_classifier
node_other_data -->|"reads benchmark data"| node_dataset_files

click node_search "https://github.com/xeaver/emotionclip/blob/main/preprocessing/search_youtube.py"
click node_download "https://github.com/xeaver/emotionclip/blob/main/preprocessing/download_videos.py"
click node_frames "https://github.com/xeaver/emotionclip/blob/main/preprocessing/frames_to_hdf5.py"
click node_detect "https://github.com/xeaver/emotionclip/blob/main/preprocessing/yolo_additions/detect_human.py"
click node_sentiment "https://github.com/xeaver/emotionclip/blob/main/preprocessing/get_sentiment_logits.py"
click node_neutral "https://github.com/xeaver/emotionclip/blob/main/preprocessing/get_neutral_scores.py"
click node_regroup "https://github.com/xeaver/emotionclip/blob/main/preprocessing/regroup_transcripts.py"
click node_index "https://github.com/xeaver/emotionclip/blob/main/preprocessing/create_index.py"
click node_youtube_data "https://github.com/xeaver/emotionclip/blob/main/src/datasets/youtube_v3.py"
click node_other_data "https://github.com/xeaver/emotionclip/tree/main/src/datasets"
click node_config "https://github.com/xeaver/emotionclip/blob/main/src/options.py"
click node_entry "https://github.com/xeaver/emotionclip/blob/main/main.py"
click node_model "https://github.com/xeaver/emotionclip/blob/main/src/models/base.py"
click node_trainer "https://github.com/xeaver/emotionclip/blob/main/src/engine/trainer.py"
click node_loss "https://github.com/xeaver/emotionclip/blob/main/src/models/loss.py"
click node_evaluation "https://github.com/xeaver/emotionclip/blob/main/src/engine/evaluation.py"
click node_linear "https://github.com/xeaver/emotionclip/blob/main/linear_eval.py"
click node_classifier "https://github.com/xeaver/emotionclip/blob/main/linear_eval.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_search,node_download,node_frames,node_detect,node_sentiment,node_neutral,node_regroup,node_index toneBlue
class node_youtube_data,node_other_data toneAmber
class node_config,node_entry,node_model,node_trainer,node_loss toneMint
class node_evaluation,node_linear,node_classifier toneRose
class node_researcher,node_youtube,node_dataset_files,node_checkpoint toneIndigo
```

## Components

### Data preparation

- [Find playlists](https://github.com/xeaver/emotionclip/blob/main/preprocessing/search_youtube.py): `preprocessing/search_youtube.py`
- [Fetch videos](https://github.com/xeaver/emotionclip/blob/main/preprocessing/download_videos.py): `preprocessing/download_videos.py`
- [Prepare frames](https://github.com/xeaver/emotionclip/blob/main/preprocessing/frames_to_hdf5.py): `preprocessing/frames_to_hdf5.py`
- [Detect people](https://github.com/xeaver/emotionclip/blob/main/preprocessing/yolo_additions/detect_human.py): `preprocessing/yolo_additions/detect_human.py`
- [Sentiment logits](https://github.com/xeaver/emotionclip/blob/main/preprocessing/get_sentiment_logits.py): `preprocessing/get_sentiment_logits.py`
- [Neutral scores](https://github.com/xeaver/emotionclip/blob/main/preprocessing/get_neutral_scores.py): `preprocessing/get_neutral_scores.py`
- [Regroup transcripts](https://github.com/xeaver/emotionclip/blob/main/preprocessing/regroup_transcripts.py): `preprocessing/regroup_transcripts.py`
- [Build index](https://github.com/xeaver/emotionclip/blob/main/preprocessing/create_index.py): `preprocessing/create_index.py`

### Datasets

- [YouTube dataset](https://github.com/xeaver/emotionclip/blob/main/src/datasets/youtube_v3.py): `src/datasets/youtube_v3.py`
- [Benchmark datasets](https://github.com/xeaver/emotionclip/tree/main/src/datasets): `src/datasets`

### Representation learning

- [Experiment config](https://github.com/xeaver/emotionclip/blob/main/src/options.py): `src/options.py`
- [Training entry](https://github.com/xeaver/emotionclip/blob/main/main.py): `main.py`
- [EmotionCLIP model](https://github.com/xeaver/emotionclip/blob/main/src/models/base.py): `src/models/base.py`
- [Training engine](https://github.com/xeaver/emotionclip/blob/main/src/engine/trainer.py): `src/engine/trainer.py`
- [Contrastive loss](https://github.com/xeaver/emotionclip/blob/main/src/models/loss.py): `src/models/loss.py`

### Evaluation

- [Evaluation engine](https://github.com/xeaver/emotionclip/blob/main/src/engine/evaluation.py): `src/engine/evaluation.py`
- [Linear evaluation](https://github.com/xeaver/emotionclip/blob/main/linear_eval.py): `linear_eval.py`
- [Linear classifier](https://github.com/xeaver/emotionclip/blob/main/linear_eval.py): `linear_eval.py`

### Other components

- **Researcher**
- **YouTube**
- **Prepared data**
- **Pretrained checkpoint**

## Connections

- Researcher → Find playlists: requests data
- YouTube → Find playlists: provides listings
- YouTube → Fetch videos: provides media
- Find playlists → Prepared data: writes metadata
- Fetch videos → Prepared data: writes video and transcripts
- Prepare frames → Prepared data: writes frame data
- Detect people → Prepared data: writes human boxes
- Sentiment logits → Prepared data: writes logits
- Neutral scores → Prepared data: writes scores
- Regroup transcripts → Prepared data: writes transcripts
- Build index → Prepared data: writes index
- Researcher → Training entry: starts training
- Training entry → Experiment config: configures
- Training entry → YouTube dataset: constructs dataset
- Prepared data → YouTube dataset: read by
- Training entry → EmotionCLIP model: creates
- Pretrained checkpoint → EmotionCLIP model: initializes weights
- Training entry → Training engine: starts training
- Training engine → EmotionCLIP model: runs forward pass
- Training engine → Contrastive loss: computes objective
- Training entry → Evaluation engine: configures evaluation
- Researcher → Linear evaluation: requests evaluation
- Linear evaluation → Benchmark datasets: constructs datasets
- Linear evaluation → EmotionCLIP model: extracts features
- Linear evaluation → Linear classifier: fits and scores
- Benchmark datasets → Prepared data: reads benchmark data