# siplysagari/ddvc architecture

> Architecture diagram of the GitHub repository siplysagari/ddvc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/siplysagari/ddvc
- Repository: https://github.com/siplysagari/ddvc

## Overview

DDVC is an end-to-end dense video captioning research system. Its principal workflow is to prepare video features and annotations, train or run a model that localizes events and generates captions, then evaluate predictions with captioning, temporal-localization, and SODA metrics. The README documents query decomposition, joint localization-and-caption assignment, and contrastive visual-text alignment; the model implementation imports its transformer, matcher, criterion, encoder, and caption-head components. Some wiring and evaluation entry-point details are not sampled, so uncertain connections are omitted.

Main entry points: [`ddvc/ddvc_ret_encdec_clip_capQ.py`](https://github.com/siplysagari/ddvc/blob/master/ddvc/ddvc_ret_encdec_clip_capQ.py) (DDVC model), [`eval.py`](https://github.com/siplysagari/ddvc/blob/master/eval.py) (Evaluation entry), [`densevid_eval3/evaluate2021.py`](https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/evaluate2021.py) (Dense caption metrics), [`train.py`](https://github.com/siplysagari/ddvc/blob/master/train.py) (Training entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Video data"]
  node_video_data["Video dataset<br/>[video_dataset.py]"]
end

subgraph group_model["Captioning model"]
  node_ddvc_model["DDVC model"]
  node_encoder["Video encoder<br/>[base_encoder.py]"]
  node_position["Position encoding"]
  node_transformer["Deformable transformer"]
  node_matcher["Joint matcher<br/>[matcher.py]"]
  node_criterion["Training criterion<br/>[criterion.py]"]
  node_caption_head["Caption heads<br/>[LSTM_DSA.py]"]
  node_deform_ops["Deformable attention ops<br/>[ms_deform_attn.py]"]
end

subgraph group_runtime["Training and inference"]
  node_train["Training entry<br/>[train.py]"]
  node_infer["Evaluation entry<br/>[eval.py]"]
  node_predictions["Caption predictions"]
end

subgraph group_evaluation["Evaluation"]
  node_dvc_eval["Dense caption metrics<br/>[evaluate2021.py]"]
  node_soda_eval["SODA evaluation<br/>[eval_soda.py]"]
  node_grounding_eval["Grounding evaluation<br/>[eval_grounding.py]"]
  node_temporal_eval["Temporal evaluation<br/>[eval_tal.py]"]
  node_cider["CIDEr scorer<br/>[cider.py]"]
  node_bleu["BLEU scorer<br/>[bleu.py]"]
  node_meteor["METEOR scorer<br/>[meteor.py]"]
  node_rouge["ROUGE scorer<br/>[rouge.py]"]
end

node_researcher(("Researcher"))

node_researcher -->|"starts training"| node_train
node_researcher -->|"runs evaluation"| node_infer
node_train -.->|"loads data"| node_video_data
node_train -.->|"trains model"| node_ddvc_model
node_infer -.->|"runs model"| node_ddvc_model
node_ddvc_model -->|"uses encoder"| node_encoder
node_ddvc_model -->|"uses encoding"| node_position
node_ddvc_model -->|"uses transformer"| node_transformer
node_ddvc_model -->|"uses matcher"| node_matcher
node_ddvc_model -->|"uses criterion"| node_criterion
node_ddvc_model -->|"uses caption head"| node_caption_head
node_caption_head -->|"uses attention"| node_deform_ops
node_ddvc_model -.->|"produces captions"| node_predictions
node_infer -.->|"scores captions"| node_dvc_eval
node_infer -.->|"scores SODA"| node_soda_eval
node_infer -.->|"scores grounding"| node_grounding_eval
node_infer -.->|"scores timing"| node_temporal_eval
node_dvc_eval -->|"uses scorer"| node_cider
node_dvc_eval -->|"uses scorer"| node_bleu
node_dvc_eval -->|"uses scorer"| node_meteor
node_dvc_eval -->|"uses scorer"| node_rouge

click node_video_data "https://github.com/siplysagari/ddvc/blob/master/data/video_dataset.py"
click node_train "https://github.com/siplysagari/ddvc/blob/master/train.py"
click node_infer "https://github.com/siplysagari/ddvc/blob/master/eval.py"
click node_ddvc_model "https://github.com/siplysagari/ddvc/blob/master/ddvc/ddvc_ret_encdec_clip_capQ.py"
click node_encoder "https://github.com/siplysagari/ddvc/blob/master/ddvc/base_encoder.py"
click node_position "https://github.com/siplysagari/ddvc/blob/master/ddvc/position_encoding.py"
click node_transformer "https://github.com/siplysagari/ddvc/blob/master/ddvc/deformable_transformer.py"
click node_matcher "https://github.com/siplysagari/ddvc/blob/master/ddvc/matcher.py"
click node_criterion "https://github.com/siplysagari/ddvc/blob/master/ddvc/criterion.py"
click node_caption_head "https://github.com/siplysagari/ddvc/blob/master/ddvc/CaptioningHead/LSTM_DSA.py"
click node_deform_ops "https://github.com/siplysagari/ddvc/blob/master/ddvc/ops/modules/ms_deform_attn.py"
click node_dvc_eval "https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/evaluate2021.py"
click node_soda_eval "https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/eval_soda.py"
click node_grounding_eval "https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/eval_grounding.py"
click node_temporal_eval "https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/eval_tal.py"
click node_cider "https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/pycocoevalcap/cider/cider.py"
click node_bleu "https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/pycocoevalcap/bleu/bleu.py"
click node_meteor "https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/pycocoevalcap/meteor/meteor.py"
click node_rouge "https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/pycocoevalcap/rouge/rouge.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_video_data toneBlue
class node_ddvc_model,node_encoder,node_position,node_transformer,node_matcher,node_criterion,node_caption_head,node_deform_ops toneAmber
class node_train,node_infer,node_predictions toneMint
class node_dvc_eval,node_soda_eval,node_grounding_eval,node_temporal_eval,node_cider,node_bleu,node_meteor,node_rouge toneRose
class node_researcher toneIndigo
```

## Components

### Video data

- [Video dataset](https://github.com/siplysagari/ddvc/blob/master/data/video_dataset.py): `data/video_dataset.py`

### Captioning model

- [DDVC model](https://github.com/siplysagari/ddvc/blob/master/ddvc/ddvc_ret_encdec_clip_capQ.py): `ddvc/ddvc_ret_encdec_clip_capQ.py`
- [Video encoder](https://github.com/siplysagari/ddvc/blob/master/ddvc/base_encoder.py): `ddvc/base_encoder.py`
- [Position encoding](https://github.com/siplysagari/ddvc/blob/master/ddvc/position_encoding.py): `ddvc/position_encoding.py`
- [Deformable transformer](https://github.com/siplysagari/ddvc/blob/master/ddvc/deformable_transformer.py): `ddvc/deformable_transformer.py`
- [Joint matcher](https://github.com/siplysagari/ddvc/blob/master/ddvc/matcher.py): `ddvc/matcher.py`
- [Training criterion](https://github.com/siplysagari/ddvc/blob/master/ddvc/criterion.py): `ddvc/criterion.py`
- [Caption heads](https://github.com/siplysagari/ddvc/blob/master/ddvc/CaptioningHead/LSTM_DSA.py): `ddvc/CaptioningHead/LSTM_DSA.py`
- [Deformable attention ops](https://github.com/siplysagari/ddvc/blob/master/ddvc/ops/modules/ms_deform_attn.py): `ddvc/ops/modules/ms_deform_attn.py`

### Training and inference

- [Training entry](https://github.com/siplysagari/ddvc/blob/master/train.py): `train.py`
- [Evaluation entry](https://github.com/siplysagari/ddvc/blob/master/eval.py): `eval.py`
- **Caption predictions**

### Evaluation

- [Dense caption metrics](https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/evaluate2021.py): `densevid_eval3/evaluate2021.py`
- [SODA evaluation](https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/eval_soda.py): `densevid_eval3/eval_soda.py`
- [Grounding evaluation](https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/eval_grounding.py): `densevid_eval3/eval_grounding.py`
- [Temporal evaluation](https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/eval_tal.py): `densevid_eval3/eval_tal.py`
- [CIDEr scorer](https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/pycocoevalcap/cider/cider.py): `densevid_eval3/pycocoevalcap/cider/cider.py`
- [BLEU scorer](https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/pycocoevalcap/bleu/bleu.py): `densevid_eval3/pycocoevalcap/bleu/bleu.py`
- [METEOR scorer](https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/pycocoevalcap/meteor/meteor.py): `densevid_eval3/pycocoevalcap/meteor/meteor.py`
- [ROUGE scorer](https://github.com/siplysagari/ddvc/blob/master/densevid_eval3/pycocoevalcap/rouge/rouge.py): `densevid_eval3/pycocoevalcap/rouge/rouge.py`

### Other components

- **Researcher**

## Connections

- Researcher → Training entry: starts training
- Researcher → Evaluation entry: runs evaluation
- Training entry → Video dataset: loads data
- Training entry → DDVC model: trains model
- Evaluation entry → DDVC model: runs model
- DDVC model → Video encoder: uses encoder (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- DDVC model → Position encoding: uses encoding (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- DDVC model → Deformable transformer: uses transformer (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- DDVC model → Joint matcher: uses matcher (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- DDVC model → Training criterion: uses criterion (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- DDVC model → Caption heads: uses caption head (evidence: `ddvc/ddvc_ret_encdec_clip_capQ.py`)
- Caption heads → Deformable attention ops: uses attention (evidence: `ddvc/CaptioningHead/LSTM_DSA.py`)
- DDVC model → Caption predictions: produces captions
- Evaluation entry → Dense caption metrics: scores captions
- Evaluation entry → SODA evaluation: scores SODA
- Evaluation entry → Grounding evaluation: scores grounding
- Evaluation entry → Temporal evaluation: scores timing
- Dense caption metrics → CIDEr scorer: uses scorer (evidence: `densevid_eval3/evaluate2021.py`)
- Dense caption metrics → BLEU scorer: uses scorer (evidence: `densevid_eval3/evaluate2021.py`)
- Dense caption metrics → METEOR scorer: uses scorer (evidence: `densevid_eval3/evaluate2021.py`)
- Dense caption metrics → ROUGE scorer: uses scorer (evidence: `densevid_eval3/evaluate2021.py`)
