# lijiaxin0111/open-d4rt architecture

> Architecture diagram of the GitHub repository lijiaxin0111/open-d4rt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lijiaxin0111/open-d4rt
- Repository: https://github.com/lijiaxin0111/open-d4rt

## Overview

OpenD4RT implements D4RT video-based 4D reconstruction and tracking, with training, WorldTrack evaluation, and visualization workflows. The principal inference path takes a video and pixel/time queries, runs the model, and produces 3D tracks or scene points; evaluation aligns and scores those predictions. The graph also shows the training data/supervision path and demo-building tools. Some source excerpts are partial, so unsampled wiring is included only where the README documents the workflow.

Main entry points: [`src/model/d4rt.py`](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/d4rt.py) (D4RT model), [`src/engine/trainer.py`](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/engine/trainer.py) (Trainer), [`eval_track3d_in_worldtrack.py`](https://github.com/lijiaxin0111/open-d4rt/blob/main/eval_track3d_in_worldtrack.py) (WorldTrack evaluator), [`src/model/builder.py`](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/builder.py) (Model builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data and supervision"]
  node_dataset_adapters["Dataset adapters"]
  node_data_builder["Dataset builder<br/>[builder.py]"]
  node_query_supervision["Query supervision"]
  node_query_sampler["Query sampling<br/>[query_sampler.py]"]
  node_kubric_preprocess["Kubric preprocessing"]
end

subgraph group_model["D4RT model"]
  node_model_builder["Model builder<br/>[builder.py]"]
  node_d4rt["D4RT model<br/>[d4rt.py]"]
  node_encoder["Video encoder<br/>[encoder.py]"]
  node_query_embedding["Query embedding<br/>[query_embedding.py]"]
  node_decoder["Query decoder<br/>[decoder.py]"]
  node_prediction_heads["Prediction heads<br/>[heads.py]"]
end

subgraph group_training["Training"]
  node_train_entry["Training entry<br/>[train.py]"]
  node_trainer["Trainer<br/>[trainer.py]"]
  node_loss["Multitask loss<br/>[d4rt_loss.py]"]
end

subgraph group_evaluation["WorldTrack evaluation"]
  node_worldtrack_eval["WorldTrack evaluator"]
  node_eval_tasks["Query and alignment<br/>[tasks.py]"]
end

subgraph group_visualization["Visualization"]
  node_demo_builder["Demo builder<br/>[build_like_demo.py]"]
  node_worldtrack_demo["WorldTrack demo"]
  node_viser["Viser viewer"]
  node_training_visualizer["Training visualizer"]
end

node_user(("Researcher"))
node_video_data[("Video datasets")]
node_worldtrack_data[("WorldTrack sequences")]
node_eval_results["Tracking metrics"]
node_demo_package["Demo assets"]

node_user -->|"starts training"| node_train_entry
node_video_data -->|"provides samples"| node_dataset_adapters
node_dataset_adapters -->|"loads datasets"| node_data_builder
node_data_builder -->|"builds targets"| node_query_supervision
node_data_builder -->|"samples queries"| node_query_sampler
node_kubric_preprocess -.->|"materializes Kubric"| node_video_data
node_train_entry -->|"starts loop"| node_trainer
node_trainer -->|"requests batches"| node_data_builder
node_trainer -->|"builds model"| node_model_builder
node_model_builder -->|"constructs"| node_d4rt
node_d4rt -->|"encodes video"| node_encoder
node_d4rt -->|"embeds queries"| node_query_embedding
node_d4rt -->|"decodes queries"| node_decoder
node_d4rt -->|"predicts outputs"| node_prediction_heads
node_trainer -->|"computes loss"| node_loss
node_trainer -->|"logs visualizations"| node_training_visualizer
node_user -->|"runs evaluation"| node_worldtrack_eval
node_worldtrack_data -->|"supplies sequences"| node_worldtrack_eval
node_worldtrack_eval -->|"loads checkpoint model"| node_model_builder
node_worldtrack_eval -->|"queries and aligns"| node_eval_tasks
node_worldtrack_eval -->|"writes metrics"| node_eval_results
node_user -->|"builds demo"| node_demo_builder
node_demo_builder -->|"loads model"| node_model_builder
node_demo_builder -->|"exports assets"| node_demo_package
node_user -->|"builds tracking demo"| node_worldtrack_demo
node_worldtrack_demo -->|"loads model"| node_model_builder
node_worldtrack_demo -->|"reads sample"| node_worldtrack_data
node_worldtrack_demo -->|"exports assets"| node_demo_package
node_user -->|"views demo"| node_viser
node_demo_package -->|"serves assets"| node_viser

click node_dataset_adapters "https://github.com/lijiaxin0111/open-d4rt/tree/main/src/data"
click node_data_builder "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/data/builder.py"
click node_query_supervision "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/data/depth_query_builder.py"
click node_query_sampler "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/data/query_sampler.py"
click node_kubric_preprocess "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/data_preprocess/prepare_kubric_full_preprocessed.py"
click node_model_builder "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/builder.py"
click node_d4rt "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/d4rt.py"
click node_encoder "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/encoder.py"
click node_query_embedding "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/query_embedding.py"
click node_decoder "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/decoder.py"
click node_prediction_heads "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/heads.py"
click node_train_entry "https://github.com/lijiaxin0111/open-d4rt/blob/main/train.py"
click node_trainer "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/engine/trainer.py"
click node_loss "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/losses/d4rt_loss.py"
click node_worldtrack_eval "https://github.com/lijiaxin0111/open-d4rt/blob/main/eval_track3d_in_worldtrack.py"
click node_eval_tasks "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/eval/tasks.py"
click node_demo_builder "https://github.com/lijiaxin0111/open-d4rt/blob/main/vis/build_like_demo.py"
click node_worldtrack_demo "https://github.com/lijiaxin0111/open-d4rt/blob/main/vis/build_like_demo_for_worldtrack.py"
click node_viser "https://github.com/lijiaxin0111/open-d4rt/blob/main/vis/serve_demo_viser.py"
click node_training_visualizer "https://github.com/lijiaxin0111/open-d4rt/blob/main/src/vis/training_visualizer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset_adapters,node_data_builder,node_query_supervision,node_query_sampler,node_kubric_preprocess toneBlue
class node_model_builder,node_d4rt,node_encoder,node_query_embedding,node_decoder,node_prediction_heads,node_video_data,node_worldtrack_data toneAmber
class node_train_entry,node_trainer,node_loss toneMint
class node_worldtrack_eval,node_eval_tasks toneRose
class node_demo_builder,node_worldtrack_demo,node_viser,node_training_visualizer,node_user,node_eval_results,node_demo_package toneIndigo
```

## Components

### Data and supervision

- [Dataset adapters](https://github.com/lijiaxin0111/open-d4rt/tree/main/src/data): `src/data`
- [Dataset builder](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/data/builder.py): `src/data/builder.py`
- [Query supervision](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/data/depth_query_builder.py): `src/data/depth_query_builder.py`
- [Query sampling](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/data/query_sampler.py): `src/data/query_sampler.py`
- [Kubric preprocessing](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/data_preprocess/prepare_kubric_full_preprocessed.py): `src/data_preprocess/prepare_kubric_full_preprocessed.py`

### D4RT model

- [Model builder](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/builder.py): `src/model/builder.py`
- [D4RT model](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/d4rt.py): `src/model/d4rt.py`
- [Video encoder](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/encoder.py): `src/model/encoder.py`
- [Query embedding](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/query_embedding.py): `src/model/query_embedding.py`
- [Query decoder](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/decoder.py): `src/model/decoder.py`
- [Prediction heads](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/model/heads.py): `src/model/heads.py`

### Training

- [Training entry](https://github.com/lijiaxin0111/open-d4rt/blob/main/train.py): `train.py`
- [Trainer](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/engine/trainer.py): `src/engine/trainer.py`
- [Multitask loss](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/losses/d4rt_loss.py): `src/losses/d4rt_loss.py`

### WorldTrack evaluation

- [WorldTrack evaluator](https://github.com/lijiaxin0111/open-d4rt/blob/main/eval_track3d_in_worldtrack.py): `eval_track3d_in_worldtrack.py`
- [Query and alignment](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/eval/tasks.py): `src/eval/tasks.py`

### Visualization

- [Demo builder](https://github.com/lijiaxin0111/open-d4rt/blob/main/vis/build_like_demo.py): `vis/build_like_demo.py`
- [WorldTrack demo](https://github.com/lijiaxin0111/open-d4rt/blob/main/vis/build_like_demo_for_worldtrack.py): `vis/build_like_demo_for_worldtrack.py`
- [Viser viewer](https://github.com/lijiaxin0111/open-d4rt/blob/main/vis/serve_demo_viser.py): `vis/serve_demo_viser.py`
- [Training visualizer](https://github.com/lijiaxin0111/open-d4rt/blob/main/src/vis/training_visualizer.py): `src/vis/training_visualizer.py`

### Other components

- **Researcher**
- **Video datasets**
- **WorldTrack sequences**
- **Tracking metrics**
- **Demo assets**

## Connections

- Researcher → Training entry: starts training
- Video datasets → Dataset adapters: provides samples
- Dataset adapters → Dataset builder: loads datasets
- Dataset builder → Query supervision: builds targets
- Dataset builder → Query sampling: samples queries
- Kubric preprocessing → Video datasets: materializes Kubric
- Training entry → Trainer: starts loop
- Trainer → Dataset builder: requests batches
- Trainer → Model builder: builds model
- Model builder → D4RT model: constructs
- D4RT model → Video encoder: encodes video
- D4RT model → Query embedding: embeds queries
- D4RT model → Query decoder: decodes queries
- D4RT model → Prediction heads: predicts outputs
- Trainer → Multitask loss: computes loss
- Trainer → Training visualizer: logs visualizations
- Researcher → WorldTrack evaluator: runs evaluation
- WorldTrack sequences → WorldTrack evaluator: supplies sequences
- WorldTrack evaluator → Model builder: loads checkpoint model
- WorldTrack evaluator → Query and alignment: queries and aligns
- WorldTrack evaluator → Tracking metrics: writes metrics
- Researcher → Demo builder: builds demo
- Demo builder → Model builder: loads model
- Demo builder → Demo assets: exports assets
- Researcher → WorldTrack demo: builds tracking demo
- WorldTrack demo → Model builder: loads model
- WorldTrack demo → WorldTrack sequences: reads sample
- WorldTrack demo → Demo assets: exports assets
- Researcher → Viser viewer: views demo
- Demo assets → Viser viewer: serves assets