# thloc1212/ddvc architecture

> Architecture diagram of the GitHub repository thloc1212/ddvc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thloc1212/ddvc
- Repository: https://github.com/thloc1212/ddvc

## Overview

This repository is an experimental DDVC system for end-to-end dense video captioning. The documented workflow is to train on pre-extracted CLIP video features, predict localized events and captions, select/merge proposals, and evaluate captioning and temporal results. The model excerpt’s import index supports its links to the principal model components; the README documents the training recipe and evaluation metrics. Some dataset, selection, and evaluation internals were not sampled, so their connections are limited to documented relationships.

Main entry points: [`ddvc/ddvc_ret_encdec_clip_capQ.py`](https://github.com/thloc1212/ddvc/blob/master/ddvc/ddvc_ret_encdec_clip_capQ.py) (DDVC model), [`tools/molab_pipeline.py`](https://github.com/thloc1212/ddvc/blob/master/tools/molab_pipeline.py) (Molab runner), [`densevid_eval3`](https://github.com/thloc1212/ddvc/tree/master/densevid_eval3) (Evaluation suite), [`train.py`](https://github.com/thloc1212/ddvc/blob/master/train.py) (Training entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_workflow["Training workflow"]
  node_molab["Molab runner<br/>[molab_pipeline.py]"]
  node_trainer["Training entry<br/>[train.py]"]
  node_options["Run options<br/>[opts.py]"]
  node_dataset["Video dataset<br/>[video_dataset.py]"]
end

subgraph group_model["DDVC model"]
  node_ddvc["DDVC model"]
  node_encoder["Base encoder<br/>[base_encoder.py]"]
  node_transformer["Deformable transformer"]
  node_matcher["Joint matcher<br/>[matcher.py]"]
  node_criterion["Training criterion<br/>[criterion.py]"]
  node_caption["Caption heads"]
  node_proposal_mask["Proposal mask<br/>[proposal_mask.py]"]
  node_dedup["Dedup loss<br/>[dedup_loss.py]"]
end

subgraph group_selection["Proposal selection"]
  node_selection["Output selection"]
end

subgraph group_evaluation["Evaluation"]
  node_evaluation["Evaluation suite"]
  node_caption_metrics["Caption metrics<br/>[evaluate2021.py]"]
  node_soda["SODA metrics<br/>[soda.py]"]
end

node_operator(("Experimenter"))
node_features["CLIP features"]

node_operator -->|"runs recipe"| node_molab
node_molab -->|"invokes training"| node_trainer
node_molab -->|"uses options"| node_options
node_molab -.->|"loads CLIP"| node_ddvc
node_dataset -.->|"provides features"| node_features
node_trainer -.->|"loads data"| node_dataset
node_ddvc -->|"uses encoder"| node_encoder
node_ddvc -->|"uses transformer"| node_transformer
node_ddvc -->|"uses matcher"| node_matcher
node_ddvc -->|"uses criterion"| node_criterion
node_ddvc -->|"uses caption heads"| node_caption
node_ddvc -->|"uses mask"| node_proposal_mask
node_ddvc -.->|"optional loss"| node_dedup
node_ddvc -.->|"selects proposals"| node_selection
node_evaluation -.->|"includes metrics"| node_caption_metrics
node_evaluation -.->|"includes SODA"| node_soda
node_operator -.->|"evaluates results"| node_evaluation

click node_molab "https://github.com/thloc1212/ddvc/blob/master/tools/molab_pipeline.py"
click node_trainer "https://github.com/thloc1212/ddvc/blob/master/train.py"
click node_options "https://github.com/thloc1212/ddvc/blob/master/opts.py"
click node_dataset "https://github.com/thloc1212/ddvc/blob/master/data/video_dataset.py"
click node_ddvc "https://github.com/thloc1212/ddvc/blob/master/ddvc/ddvc_ret_encdec_clip_capQ.py"
click node_encoder "https://github.com/thloc1212/ddvc/blob/master/ddvc/base_encoder.py"
click node_transformer "https://github.com/thloc1212/ddvc/blob/master/ddvc/deformable_transformer.py"
click node_matcher "https://github.com/thloc1212/ddvc/blob/master/ddvc/matcher.py"
click node_criterion "https://github.com/thloc1212/ddvc/blob/master/ddvc/criterion.py"
click node_caption "https://github.com/thloc1212/ddvc/tree/master/ddvc/CaptioningHead"
click node_proposal_mask "https://github.com/thloc1212/ddvc/blob/master/ddvc/CaptioningHead/proposal_mask.py"
click node_selection "https://github.com/thloc1212/ddvc/blob/master/misc/proposal_selection.py"
click node_dedup "https://github.com/thloc1212/ddvc/blob/master/ddvc/dedup_loss.py"
click node_evaluation "https://github.com/thloc1212/ddvc/tree/master/densevid_eval3"
click node_caption_metrics "https://github.com/thloc1212/ddvc/blob/master/densevid_eval3/evaluate2021.py"
click node_soda "https://github.com/thloc1212/ddvc/blob/master/densevid_eval3/SODA/soda.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_molab,node_trainer,node_options,node_dataset toneBlue
class node_ddvc,node_encoder,node_transformer,node_matcher,node_criterion,node_caption,node_proposal_mask,node_dedup toneAmber
class node_selection toneMint
class node_evaluation,node_caption_metrics,node_soda toneRose
class node_operator,node_features toneIndigo
```

## Components

### Training workflow

- [Molab runner](https://github.com/thloc1212/ddvc/blob/master/tools/molab_pipeline.py): `tools/molab_pipeline.py`
- [Training entry](https://github.com/thloc1212/ddvc/blob/master/train.py): `train.py`
- [Run options](https://github.com/thloc1212/ddvc/blob/master/opts.py): `opts.py`
- [Video dataset](https://github.com/thloc1212/ddvc/blob/master/data/video_dataset.py): `data/video_dataset.py`

### DDVC model

- [DDVC model](https://github.com/thloc1212/ddvc/blob/master/ddvc/ddvc_ret_encdec_clip_capQ.py): `ddvc/ddvc_ret_encdec_clip_capQ.py`
- [Base encoder](https://github.com/thloc1212/ddvc/blob/master/ddvc/base_encoder.py): `ddvc/base_encoder.py`
- [Deformable transformer](https://github.com/thloc1212/ddvc/blob/master/ddvc/deformable_transformer.py): `ddvc/deformable_transformer.py`
- [Joint matcher](https://github.com/thloc1212/ddvc/blob/master/ddvc/matcher.py): `ddvc/matcher.py`
- [Training criterion](https://github.com/thloc1212/ddvc/blob/master/ddvc/criterion.py): `ddvc/criterion.py`
- [Caption heads](https://github.com/thloc1212/ddvc/tree/master/ddvc/CaptioningHead): `ddvc/CaptioningHead`
- [Proposal mask](https://github.com/thloc1212/ddvc/blob/master/ddvc/CaptioningHead/proposal_mask.py): `ddvc/CaptioningHead/proposal_mask.py`
- [Dedup loss](https://github.com/thloc1212/ddvc/blob/master/ddvc/dedup_loss.py): `ddvc/dedup_loss.py`

### Proposal selection

- [Output selection](https://github.com/thloc1212/ddvc/blob/master/misc/proposal_selection.py): `misc/proposal_selection.py`

### Evaluation

- [Evaluation suite](https://github.com/thloc1212/ddvc/tree/master/densevid_eval3): `densevid_eval3`
- [Caption metrics](https://github.com/thloc1212/ddvc/blob/master/densevid_eval3/evaluate2021.py): `densevid_eval3/evaluate2021.py`
- [SODA metrics](https://github.com/thloc1212/ddvc/blob/master/densevid_eval3/SODA/soda.py): `densevid_eval3/SODA/soda.py`

### Other components

- **Experimenter**
- **CLIP features**

## Connections

- Experimenter → Molab runner: runs recipe (evidence: `README.md`)
- Molab runner → Training entry: invokes training (evidence: `tools/molab_pipeline.py`)
- Molab runner → Run options: uses options (evidence: `tools/molab_pipeline.py`)
- Molab runner → DDVC model: loads CLIP
- Video dataset → CLIP features: provides features
- Training entry → Video dataset: loads data
- DDVC model → Base encoder: uses encoder (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- DDVC model → Deformable transformer: uses transformer (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- DDVC model → Joint matcher: uses matcher (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- DDVC model → Training criterion: uses criterion (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- DDVC model → Caption heads: uses caption heads (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- DDVC model → Proposal mask: uses mask (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- DDVC model → Dedup loss: optional loss
- DDVC model → Output selection: selects proposals
- Evaluation suite → Caption metrics: includes metrics (evidence: `densevid_eval3/evaluate2021.py`)
- Evaluation suite → SODA metrics: includes SODA
- Experimenter → Evaluation suite: evaluates results