How darshansingh11/avlectures works
This research pipeline prepares AVLectures data, extracts multimodal features, learns lecture-aware embeddings, and clusters those embeddings into temporal segments. The README documents the training, optional fine-tuning, feature extraction, and TW-FINCH stages; sampled source also shows subtitle processing, feature loading, and temporal clustering. Some README-described capabilities are represented at their enclosing directory because their implementation was not sampled, so uncertain internal wiring is omitted.
Main entry points: train.py Train and fine-tune · video_feature_extractor Visual feature extraction · main.py TW-FINCH clustering · model.py Joint embedding model
Components 17
Lecture preparation
- Subtitle parsing
code/helpers/Split_Videos/parse_subtitles.py - Video splitting
code/helpers/Split_Videos/driver.py - Intro trimming
code/helpers/Trim_Intro/cut_intro.py
Feature extraction
- Visual feature extraction
code/lecture_aware_embds/video_feature_extractor - Video CNN
code/lecture_aware_embds/video_feature_extractor/videocnn/main.py - Multimodal feature records
code/lecture_aware_embds/video_feature_extractor/create_pickle_ocr.py
Embedding learning
- Training data loader
code/lecture_aware_embds/avlectures_dataloader.py - Joint embedding model
code/lecture_aware_embds/model.py - Embedding objectives
code/lecture_aware_embds/loss.py - Train and fine-tune
code/lecture_aware_embds/train.py - Model evaluation
code/lecture_aware_embds/eval.py - Lecture embeddings
code/lecture_aware_embds/extract_feats.py
Temporal segmentation
- TW-FINCH clustering
code/TW_FINCH/main.py - FINCH algorithm
code/TW_FINCH/finch.py - Predicted clusters
Other components
- Researcher
- AVLectures data
Connections 19
- Researcher → AVLectures data: downloads courses
- AVLectures data → Subtitle parsing: provides subtitles
- Subtitle parsing → Video splitting: parsed segments
- AVLectures data → Intro trimming: provides lectures
- Video splitting → Visual feature extraction: segment videos
- Intro trimming → Visual feature extraction: processed videos
- Visual feature extraction → Video CNN: extracts visual features
- Visual feature extraction → Multimodal feature records: supplies features
- Multimodal feature records → Training data loader: feature records
- Training data loader → Train and fine-tune: training batches
- Train and fine-tune → Joint embedding model: optimizes
- Train and fine-tune → Embedding objectives: computes objectives
- Train and fine-tune → Model evaluation: evaluates model
- Joint embedding model → Model evaluation: model outputs
- Joint embedding model → Lecture embeddings: encodes lectures
- Lecture embeddings → TW-FINCH clustering: learned embeddings
- TW-FINCH clustering → FINCH algorithm: calls clustering
- FINCH algorithm → TW-FINCH clustering: returns labels
- TW-FINCH clustering → Predicted clusters: writes clusters