# darshansingh11/avlectures architecture

> Architecture diagram of the GitHub repository darshansingh11/avlectures, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/darshansingh11/avlectures
- Repository: https://github.com/darshansingh11/avlectures

## Overview

This research pipeline prepares AVLectures data, extracts multimodal features, learns lecture-aware embeddings, and clusters those embeddings into temporal segments. The README documents the training, optional fine-tuning, feature extraction, and TW-FINCH stages; sampled source also shows subtitle processing, feature loading, and temporal clustering. Some README-described capabilities are represented at their enclosing directory because their implementation was not sampled, so uncertain internal wiring is omitted.

Main entry points: [`code/lecture_aware_embds/train.py`](https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/train.py) (Train and fine-tune), [`code/lecture_aware_embds/video_feature_extractor`](https://github.com/darshansingh11/avlectures/tree/main/code/lecture_aware_embds/video_feature_extractor) (Visual feature extraction), [`code/TW_FINCH/main.py`](https://github.com/darshansingh11/avlectures/blob/main/code/TW_FINCH/main.py) (TW-FINCH clustering), [`code/lecture_aware_embds/model.py`](https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/model.py) (Joint embedding model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_preparation["Lecture preparation"]
  node_subtitle_parse["Subtitle parsing<br/>[parse_subtitles.py]"]
  node_split_driver["Video splitting<br/>[driver.py]"]
  node_intro_trim["Intro trimming<br/>[cut_intro.py]"]
end

subgraph group_features["Feature extraction"]
  node_video_features["Visual feature extraction"]
  node_cnn_classifier["Video CNN<br/>[main.py]"]
  node_feature_records["Multimodal feature records"]
end

subgraph group_embeddings["Embedding learning"]
  node_dataset_loader["Training data loader"]
  node_embedding_model["Joint embedding model<br/>[model.py]"]
  node_training_losses["Embedding objectives<br/>[loss.py]"]
  node_train["Train and fine-tune<br/>[train.py]"]
  node_evaluation["Model evaluation<br/>[eval.py]"]
  node_embedding_extract["Lecture embeddings<br/>[extract_feats.py]"]
end

subgraph group_segmentation["Temporal segmentation"]
  node_cluster_driver["TW-FINCH clustering<br/>[main.py]"]
  node_finch["FINCH algorithm<br/>[finch.py]"]
  node_segment_clusters["Predicted clusters"]
end

node_researcher(("Researcher"))
node_avlectures[("AVLectures data")]

node_researcher -->|"downloads courses"| node_avlectures
node_avlectures -->|"provides subtitles"| node_subtitle_parse
node_subtitle_parse -->|"parsed segments"| node_split_driver
node_avlectures -->|"provides lectures"| node_intro_trim
node_split_driver -->|"segment videos"| node_video_features
node_intro_trim -->|"processed videos"| node_video_features
node_video_features -->|"extracts visual features"| node_cnn_classifier
node_video_features -->|"supplies features"| node_feature_records
node_feature_records -->|"feature records"| node_dataset_loader
node_dataset_loader -->|"training batches"| node_train
node_train -->|"optimizes"| node_embedding_model
node_train -->|"computes objectives"| node_training_losses
node_train -->|"evaluates model"| node_evaluation
node_embedding_model -->|"model outputs"| node_evaluation
node_embedding_model -->|"encodes lectures"| node_embedding_extract
node_embedding_extract -->|"learned embeddings"| node_cluster_driver
node_cluster_driver -->|"calls clustering"| node_finch
node_finch -->|"returns labels"| node_cluster_driver
node_cluster_driver -->|"writes clusters"| node_segment_clusters

click node_subtitle_parse "https://github.com/darshansingh11/avlectures/blob/main/code/helpers/Split_Videos/parse_subtitles.py"
click node_split_driver "https://github.com/darshansingh11/avlectures/blob/main/code/helpers/Split_Videos/driver.py"
click node_intro_trim "https://github.com/darshansingh11/avlectures/blob/main/code/helpers/Trim_Intro/cut_intro.py"
click node_video_features "https://github.com/darshansingh11/avlectures/tree/main/code/lecture_aware_embds/video_feature_extractor"
click node_cnn_classifier "https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/video_feature_extractor/videocnn/main.py"
click node_feature_records "https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/video_feature_extractor/create_pickle_ocr.py"
click node_dataset_loader "https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/avlectures_dataloader.py"
click node_embedding_model "https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/model.py"
click node_training_losses "https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/loss.py"
click node_train "https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/train.py"
click node_evaluation "https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/eval.py"
click node_embedding_extract "https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/extract_feats.py"
click node_cluster_driver "https://github.com/darshansingh11/avlectures/blob/main/code/TW_FINCH/main.py"
click node_finch "https://github.com/darshansingh11/avlectures/blob/main/code/TW_FINCH/finch.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_subtitle_parse,node_split_driver,node_intro_trim toneBlue
class node_video_features,node_cnn_classifier,node_feature_records,node_avlectures toneAmber
class node_dataset_loader,node_embedding_model,node_training_losses,node_train,node_evaluation,node_embedding_extract toneMint
class node_cluster_driver,node_finch,node_segment_clusters toneRose
class node_researcher toneIndigo
```

## Components

### Lecture preparation

- [Subtitle parsing](https://github.com/darshansingh11/avlectures/blob/main/code/helpers/Split_Videos/parse_subtitles.py): `code/helpers/Split_Videos/parse_subtitles.py`
- [Video splitting](https://github.com/darshansingh11/avlectures/blob/main/code/helpers/Split_Videos/driver.py): `code/helpers/Split_Videos/driver.py`
- [Intro trimming](https://github.com/darshansingh11/avlectures/blob/main/code/helpers/Trim_Intro/cut_intro.py): `code/helpers/Trim_Intro/cut_intro.py`

### Feature extraction

- [Visual feature extraction](https://github.com/darshansingh11/avlectures/tree/main/code/lecture_aware_embds/video_feature_extractor): `code/lecture_aware_embds/video_feature_extractor`
- [Video CNN](https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/video_feature_extractor/videocnn/main.py): `code/lecture_aware_embds/video_feature_extractor/videocnn/main.py`
- [Multimodal feature records](https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/video_feature_extractor/create_pickle_ocr.py): `code/lecture_aware_embds/video_feature_extractor/create_pickle_ocr.py`

### Embedding learning

- [Training data loader](https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/avlectures_dataloader.py): `code/lecture_aware_embds/avlectures_dataloader.py`
- [Joint embedding model](https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/model.py): `code/lecture_aware_embds/model.py`
- [Embedding objectives](https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/loss.py): `code/lecture_aware_embds/loss.py`
- [Train and fine-tune](https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/train.py): `code/lecture_aware_embds/train.py`
- [Model evaluation](https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/eval.py): `code/lecture_aware_embds/eval.py`
- [Lecture embeddings](https://github.com/darshansingh11/avlectures/blob/main/code/lecture_aware_embds/extract_feats.py): `code/lecture_aware_embds/extract_feats.py`

### Temporal segmentation

- [TW-FINCH clustering](https://github.com/darshansingh11/avlectures/blob/main/code/TW_FINCH/main.py): `code/TW_FINCH/main.py`
- [FINCH algorithm](https://github.com/darshansingh11/avlectures/blob/main/code/TW_FINCH/finch.py): `code/TW_FINCH/finch.py`
- **Predicted clusters**

### Other components

- **Researcher**
- **AVLectures data**

## Connections

- Researcher → AVLectures data: downloads courses
- AVLectures data → Subtitle parsing: provides subtitles
- Subtitle parsing → Video splitting: parsed segments
- AVLectures data → Intro trimming: provides lectures
- Video splitting → Visual feature extraction: segment videos
- Intro trimming → Visual feature extraction: processed videos
- Visual feature extraction → Video CNN: extracts visual features
- Visual feature extraction → Multimodal feature records: supplies features
- Multimodal feature records → Training data loader: feature records
- Training data loader → Train and fine-tune: training batches
- Train and fine-tune → Joint embedding model: optimizes
- Train and fine-tune → Embedding objectives: computes objectives
- Train and fine-tune → Model evaluation: evaluates model
- Joint embedding model → Model evaluation: model outputs
- Joint embedding model → Lecture embeddings: encodes lectures
- Lecture embeddings → TW-FINCH clustering: learned embeddings
- TW-FINCH clustering → FINCH algorithm: calls clustering
- FINCH algorithm → TW-FINCH clustering: returns labels
- TW-FINCH clustering → Predicted clusters: writes clusters