# uzh-rpg/rvt architecture

> Architecture diagram of the GitHub repository uzh-rpg/rvt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uzh-rpg/rvt
- Repository: https://github.com/uzh-rpg/rvt

## Overview

RVT trains and evaluates recurrent object detectors from Gen1/Gen4 event-camera sequences: data loading and event representation feed a stateful MaxViT-RNN/YOLOX detector, whose predictions are measured with Prophesee metrics and optionally visualized/logged. The README also documents dataset preprocessing and streamed or random sampling. The graph follows documented workflows and sampled call sites; several internal dataset-to-loader and detector-to-head links are inferred only where module responsibilities are explicit, so unsampled wiring is omitted.

Main entry points: [`modules/data/genx.py`](https://github.com/uzh-rpg/rvt/blob/master/modules/data/genx.py) (GenX data module), [`modules/detection.py`](https://github.com/uzh-rpg/rvt/blob/master/modules/detection.py) (Detection module), [`data/genx_utils/dataset_streaming.py`](https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/dataset_streaming.py) (Stream sampling), [`data/genx_utils/dataset_rnd.py`](https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/dataset_rnd.py) (Random sampling)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Event data"]
  node_preprocess["Dataset preprocessing"]
  node_datamodule["GenX data module<br/>[genx.py]"]
  node_sequences["Sequence datasets<br/>[sequence_base.py]"]
  node_streaming["Stream sampling"]
  node_random["Random sampling<br/>[dataset_rnd.py]"]
  node_collation["Batch collation<br/>[collate.py]"]
  node_representation["Event representation<br/>[representations.py]"]
  node_augmentation["Spatial augmentation<br/>[augmentor.py]"]
  node_labels["Object labels<br/>[labels.py]"]
end

subgraph group_model["Detection model"]
  node_backbone["MaxViT-RNN backbone<br/>[maxvit_rnn.py]"]
  node_detector["YOLOX detector<br/>[detector.py]"]
  node_head["YOLOX head<br/>[yolo_head.py]"]
end

subgraph group_workflow["Training and evaluation"]
  node_train["Training entry<br/>[train.py]"]
  node_validation["Validation entry<br/>[validation.py]"]
  node_detmodule["Detection module<br/>[detection.py]"]
  node_state["Recurrent state helpers<br/>[detection.py]"]
end

subgraph group_observability["Metrics and logging"]
  node_evaluation["Prophesee evaluation<br/>[evaluator.py]"]
  node_callbacks["Detection callbacks<br/>[detection.py]"]
  node_logger["W&amp;B logger<br/>[wandb_logger.py]"]
end

node_user(("Researcher"))
node_wandb(("Weights &amp; Biases"))

node_user -->|"prepares data"| node_preprocess
node_user -->|"starts training"| node_train
node_user -->|"starts evaluation"| node_validation
node_preprocess -->|"creates dataset files"| node_sequences
node_train -->|"configures loading"| node_datamodule
node_validation -->|"configures loading"| node_datamodule
node_datamodule -->|"builds stream dataset"| node_streaming
node_datamodule -->|"builds random dataset"| node_random
node_streaming -->|"reads sequences"| node_sequences
node_random -->|"reads sequences"| node_sequences
node_datamodule -->|"selects collator"| node_collation
node_streaming -->|"forms event tensors"| node_representation
node_random -->|"forms event tensors"| node_representation
node_streaming -->|"loads labels"| node_labels
node_random -->|"loads labels"| node_labels
node_augmentation -->|"transforms boxes"| node_labels
node_datamodule -->|"provides batches"| node_train
node_datamodule -->|"provides batches"| node_validation
node_train -->|"trains module"| node_detmodule
node_validation -->|"evaluates module"| node_detmodule
node_detmodule -->|"runs detector"| node_detector
node_detmodule -->|"manages recurrent state"| node_state
node_detector -->|"extracts features"| node_backbone
node_detector -->|"predicts boxes"| node_head
node_detmodule -->|"submits predictions"| node_evaluation
node_detmodule -->|"emits visualization hooks"| node_callbacks
node_callbacks -->|"logs visualizations"| node_logger
node_logger -->|"uploads metrics"| node_wandb

click node_preprocess "https://github.com/uzh-rpg/rvt/blob/master/scripts/genx/preprocess_dataset.py"
click node_datamodule "https://github.com/uzh-rpg/rvt/blob/master/modules/data/genx.py"
click node_sequences "https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/sequence_base.py"
click node_streaming "https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/dataset_streaming.py"
click node_random "https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/dataset_rnd.py"
click node_collation "https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/collate.py"
click node_representation "https://github.com/uzh-rpg/rvt/blob/master/data/utils/representations.py"
click node_augmentation "https://github.com/uzh-rpg/rvt/blob/master/data/utils/augmentor.py"
click node_labels "https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/labels.py"
click node_train "https://github.com/uzh-rpg/rvt/blob/master/train.py"
click node_validation "https://github.com/uzh-rpg/rvt/blob/master/validation.py"
click node_detmodule "https://github.com/uzh-rpg/rvt/blob/master/modules/detection.py"
click node_state "https://github.com/uzh-rpg/rvt/blob/master/modules/utils/detection.py"
click node_backbone "https://github.com/uzh-rpg/rvt/blob/master/models/detection/recurrent_backbone/maxvit_rnn.py"
click node_detector "https://github.com/uzh-rpg/rvt/blob/master/models/detection/yolox_extension/models/detector.py"
click node_head "https://github.com/uzh-rpg/rvt/blob/master/models/detection/yolox/models/yolo_head.py"
click node_evaluation "https://github.com/uzh-rpg/rvt/blob/master/utils/evaluation/prophesee/evaluator.py"
click node_callbacks "https://github.com/uzh-rpg/rvt/blob/master/callbacks/detection.py"
click node_logger "https://github.com/uzh-rpg/rvt/blob/master/loggers/wandb_logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_preprocess,node_datamodule,node_sequences,node_streaming,node_random,node_collation,node_representation,node_augmentation,node_labels toneBlue
class node_backbone,node_detector,node_head toneAmber
class node_train,node_validation,node_detmodule,node_state toneMint
class node_evaluation,node_callbacks,node_logger toneRose
class node_user,node_wandb toneIndigo
```

## Components

### Event data

- [Dataset preprocessing](https://github.com/uzh-rpg/rvt/blob/master/scripts/genx/preprocess_dataset.py): `scripts/genx/preprocess_dataset.py`
- [GenX data module](https://github.com/uzh-rpg/rvt/blob/master/modules/data/genx.py): `modules/data/genx.py`
- [Sequence datasets](https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/sequence_base.py): `data/genx_utils/sequence_base.py`
- [Stream sampling](https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/dataset_streaming.py): `data/genx_utils/dataset_streaming.py`
- [Random sampling](https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/dataset_rnd.py): `data/genx_utils/dataset_rnd.py`
- [Batch collation](https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/collate.py): `data/genx_utils/collate.py`
- [Event representation](https://github.com/uzh-rpg/rvt/blob/master/data/utils/representations.py): `data/utils/representations.py`
- [Spatial augmentation](https://github.com/uzh-rpg/rvt/blob/master/data/utils/augmentor.py): `data/utils/augmentor.py`
- [Object labels](https://github.com/uzh-rpg/rvt/blob/master/data/genx_utils/labels.py): `data/genx_utils/labels.py`

### Detection model

- [MaxViT-RNN backbone](https://github.com/uzh-rpg/rvt/blob/master/models/detection/recurrent_backbone/maxvit_rnn.py): `models/detection/recurrent_backbone/maxvit_rnn.py`
- [YOLOX detector](https://github.com/uzh-rpg/rvt/blob/master/models/detection/yolox_extension/models/detector.py): `models/detection/yolox_extension/models/detector.py`
- [YOLOX head](https://github.com/uzh-rpg/rvt/blob/master/models/detection/yolox/models/yolo_head.py): `models/detection/yolox/models/yolo_head.py`

### Training and evaluation

- [Training entry](https://github.com/uzh-rpg/rvt/blob/master/train.py): `train.py`
- [Validation entry](https://github.com/uzh-rpg/rvt/blob/master/validation.py): `validation.py`
- [Detection module](https://github.com/uzh-rpg/rvt/blob/master/modules/detection.py): `modules/detection.py`
- [Recurrent state helpers](https://github.com/uzh-rpg/rvt/blob/master/modules/utils/detection.py): `modules/utils/detection.py`

### Metrics and logging

- [Prophesee evaluation](https://github.com/uzh-rpg/rvt/blob/master/utils/evaluation/prophesee/evaluator.py): `utils/evaluation/prophesee/evaluator.py`
- [Detection callbacks](https://github.com/uzh-rpg/rvt/blob/master/callbacks/detection.py): `callbacks/detection.py`
- [W&B logger](https://github.com/uzh-rpg/rvt/blob/master/loggers/wandb_logger.py): `loggers/wandb_logger.py`

### Other components

- **Researcher**
- **Weights & Biases**

## Connections

- Researcher → Dataset preprocessing: prepares data
- Researcher → Training entry: starts training
- Researcher → Validation entry: starts evaluation
- Dataset preprocessing → Sequence datasets: creates dataset files
- Training entry → GenX data module: configures loading
- Validation entry → GenX data module: configures loading
- GenX data module → Stream sampling: builds stream dataset
- GenX data module → Random sampling: builds random dataset
- Stream sampling → Sequence datasets: reads sequences
- Random sampling → Sequence datasets: reads sequences
- GenX data module → Batch collation: selects collator
- Stream sampling → Event representation: forms event tensors
- Random sampling → Event representation: forms event tensors
- Stream sampling → Object labels: loads labels
- Random sampling → Object labels: loads labels
- Spatial augmentation → Object labels: transforms boxes
- GenX data module → Training entry: provides batches
- GenX data module → Validation entry: provides batches
- Training entry → Detection module: trains module
- Validation entry → Detection module: evaluates module
- Detection module → YOLOX detector: runs detector
- Detection module → Recurrent state helpers: manages recurrent state
- YOLOX detector → MaxViT-RNN backbone: extracts features
- YOLOX detector → YOLOX head: predicts boxes
- Detection module → Prophesee evaluation: submits predictions
- Detection module → Detection callbacks: emits visualization hooks
- Detection callbacks → W&B logger: logs visualizations
- W&B logger → Weights & Biases: uploads metrics