How lijiaxin0111/open-d4rt works
OpenD4RT implements D4RT video-based 4D reconstruction and tracking, with training, WorldTrack evaluation, and visualization workflows. The principal inference path takes a video and pixel/time queries, runs the model, and produces 3D tracks or scene points; evaluation aligns and scores those predictions. The graph also shows the training data/supervision path and demo-building tools. Some source excerpts are partial, so unsampled wiring is included only where the README documents the workflow.
Main entry points: d4rt.py D4RT model · trainer.py Trainer · eval_track3d_in_worldtrack.py WorldTrack evaluator · builder.py Model builder
Components 25
Data and supervision
- Dataset adapters
src/data - Dataset builder
src/data/builder.py - Query supervision
src/data/depth_query_builder.py - Query sampling
src/data/query_sampler.py - Kubric preprocessing
src/data_preprocess/prepare_kubric_full_preprocessed.py
D4RT model
- Model builder
src/model/builder.py - D4RT model
src/model/d4rt.py - Video encoder
src/model/encoder.py - Query embedding
src/model/query_embedding.py - Query decoder
src/model/decoder.py - Prediction heads
src/model/heads.py
Training
- Training entry
train.py - Trainer
src/engine/trainer.py - Multitask loss
src/losses/d4rt_loss.py
WorldTrack evaluation
- WorldTrack evaluator
eval_track3d_in_worldtrack.py - Query and alignment
src/eval/tasks.py
Visualization
- Demo builder
vis/build_like_demo.py - WorldTrack demo
vis/build_like_demo_for_worldtrack.py - Viser viewer
vis/serve_demo_viser.py - Training visualizer
src/vis/training_visualizer.py
Other components
- Researcher
- Video datasets
- WorldTrack sequences
- Tracking metrics
- Demo assets
Connections 30
- Researcher → Training entry: starts training
- Video datasets → Dataset adapters: provides samples
- Dataset adapters → Dataset builder: loads datasets
- Dataset builder → Query supervision: builds targets
- Dataset builder → Query sampling: samples queries
- Kubric preprocessing → Video datasets: materializes Kubric
- Training entry → Trainer: starts loop
- Trainer → Dataset builder: requests batches
- Trainer → Model builder: builds model
- Model builder → D4RT model: constructs
- D4RT model → Video encoder: encodes video
- D4RT model → Query embedding: embeds queries
- D4RT model → Query decoder: decodes queries
- D4RT model → Prediction heads: predicts outputs
- Trainer → Multitask loss: computes loss
- Trainer → Training visualizer: logs visualizations
- Researcher → WorldTrack evaluator: runs evaluation
- WorldTrack sequences → WorldTrack evaluator: supplies sequences
- WorldTrack evaluator → Model builder: loads checkpoint model
- WorldTrack evaluator → Query and alignment: queries and aligns
- WorldTrack evaluator → Tracking metrics: writes metrics
- Researcher → Demo builder: builds demo
- Demo builder → Model builder: loads model
- Demo builder → Demo assets: exports assets
- Researcher → WorldTrack demo: builds tracking demo
- WorldTrack demo → Model builder: loads model
- WorldTrack demo → WorldTrack sequences: reads sample
- WorldTrack demo → Demo assets: exports assets
- Researcher → Viser viewer: views demo
- Demo assets → Viser viewer: serves assets