How siplysagari/ddvc works
DDVC is an end-to-end dense video captioning research system. Its principal workflow is to prepare video features and annotations, train or run a model that localizes events and generates captions, then evaluate predictions with captioning, temporal-localization, and SODA metrics. The README documents query decomposition, joint localization-and-caption assignment, and contrastive visual-text alignment; the model implementation imports its transformer, matcher, criterion, encoder, and caption-head components. Some wiring and evaluation entry-point details are not sampled, so uncertain connections are omitted.
Main entry points: ddvc_ret_encdec_clip_capQ.py DDVC model · eval.py Evaluation entry · evaluate2021.py Dense caption metrics · train.py Training entry
Components 21
Video data
- Video dataset
data/video_dataset.py
Captioning model
- DDVC model
ddvc/ddvc_ret_encdec_clip_capQ.py - Video encoder
ddvc/base_encoder.py - Position encoding
ddvc/position_encoding.py - Deformable transformer
ddvc/deformable_transformer.py - Joint matcher
ddvc/matcher.py - Training criterion
ddvc/criterion.py - Caption heads
ddvc/CaptioningHead/LSTM_DSA.py - Deformable attention ops
ddvc/ops/modules/ms_deform_attn.py
Training and inference
- Training entry
train.py - Evaluation entry
eval.py - Caption predictions
Evaluation
- Dense caption metrics
densevid_eval3/evaluate2021.py - SODA evaluation
densevid_eval3/eval_soda.py - Grounding evaluation
densevid_eval3/eval_grounding.py - Temporal evaluation
densevid_eval3/eval_tal.py - CIDEr scorer
densevid_eval3/pycocoevalcap/cider/cider.py - BLEU scorer
densevid_eval3/pycocoevalcap/bleu/bleu.py - METEOR scorer
densevid_eval3/pycocoevalcap/meteor/meteor.py - ROUGE scorer
densevid_eval3/pycocoevalcap/rouge/rouge.py
Other components
- Researcher
Connections 21
- Researcher → Training entry: starts training
- Researcher → Evaluation entry: runs evaluation
- Training entry → Video dataset: loads data
- Training entry → DDVC model: trains model
- Evaluation entry → DDVC model: runs model
- DDVC model → Video encoder: uses encoder
ddvc/ddvc_ret_encdec_clip_capQ.py - DDVC model → Position encoding: uses encoding
ddvc/ddvc_ret_encdec_clip_capQ.py - DDVC model → Deformable transformer: uses transformer
ddvc/ddvc_ret_encdec_clip_capQ.py - DDVC model → Joint matcher: uses matcher
ddvc/ddvc_ret_encdec_clip_capQ.py - DDVC model → Training criterion: uses criterion
ddvc/ddvc_ret_encdec_clip_capQ.py - DDVC model → Caption heads: uses caption head
ddvc/ddvc_ret_encdec_clip_capQ.py - Caption heads → Deformable attention ops: uses attention
ddvc/CaptioningHead/LSTM_DSA.py - DDVC model → Caption predictions: produces captions
- Evaluation entry → Dense caption metrics: scores captions
- Evaluation entry → SODA evaluation: scores SODA
- Evaluation entry → Grounding evaluation: scores grounding
- Evaluation entry → Temporal evaluation: scores timing
- Dense caption metrics → CIDEr scorer: uses scorer
densevid_eval3/evaluate2021.py - Dense caption metrics → BLEU scorer: uses scorer
densevid_eval3/evaluate2021.py - Dense caption metrics → METEOR scorer: uses scorer
densevid_eval3/evaluate2021.py - Dense caption metrics → ROUGE scorer: uses scorer
densevid_eval3/evaluate2021.py