How thloc1212/ddvc works
This repository is an experimental DDVC system for end-to-end dense video captioning. The documented workflow is to train on pre-extracted CLIP video features, predict localized events and captions, select/merge proposals, and evaluate captioning and temporal results. The model excerpt’s import index supports its links to the principal model components; the README documents the training recipe and evaluation metrics. Some dataset, selection, and evaluation internals were not sampled, so their connections are limited to documented relationships.
Main entry points: ddvc_ret_encdec_clip_capQ.py DDVC model · molab_pipeline.py Molab runner · densevid_eval3 Evaluation suite · train.py Training entry
Components 18
Training workflow
- Molab runner
tools/molab_pipeline.py - Training entry
train.py - Run options
opts.py - Video dataset
data/video_dataset.py
DDVC model
- DDVC model
ddvc/ddvc_ret_encdec_clip_capQ.py - Base encoder
ddvc/base_encoder.py - Deformable transformer
ddvc/deformable_transformer.py - Joint matcher
ddvc/matcher.py - Training criterion
ddvc/criterion.py - Caption heads
ddvc/CaptioningHead - Proposal mask
ddvc/CaptioningHead/proposal_mask.py - Dedup loss
ddvc/dedup_loss.py
Proposal selection
- Output selection
misc/proposal_selection.py
Evaluation
- Evaluation suite
densevid_eval3 - Caption metrics
densevid_eval3/evaluate2021.py - SODA metrics
densevid_eval3/SODA/soda.py
Other components
- Experimenter
- CLIP features
Connections 17
- Experimenter → Molab runner: runs recipe
README.md - Molab runner → Training entry: invokes training
tools/molab_pipeline.py - Molab runner → Run options: uses options
tools/molab_pipeline.py - Molab runner → DDVC model: loads CLIP
- Video dataset → CLIP features: provides features
- Training entry → Video dataset: loads data
- DDVC model → Base encoder: uses encoder
ddvc/ddvc_ret_encdec_clip_capQ.py - DDVC model → Deformable transformer: uses transformer
ddvc/ddvc_ret_encdec_clip_capQ.py - DDVC model → Joint matcher: uses matcher
ddvc/ddvc_ret_encdec_clip_capQ.py - DDVC model → Training criterion: uses criterion
ddvc/ddvc_ret_encdec_clip_capQ.py - DDVC model → Caption heads: uses caption heads
ddvc/ddvc_ret_encdec_clip_capQ.py - DDVC model → Proposal mask: uses mask
ddvc/ddvc_ret_encdec_clip_capQ.py - DDVC model → Dedup loss: optional loss
- DDVC model → Output selection: selects proposals
- Evaluation suite → Caption metrics: includes metrics
densevid_eval3/evaluate2021.py - Evaluation suite → SODA metrics: includes SODA
- Experimenter → Evaluation suite: evaluates results