# uzh-rpg/rpg_e2vid architecture

> Architecture diagram of the GitHub repository uzh-rpg/rpg_e2vid, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uzh-rpg/rpg_e2vid
- Repository: https://github.com/uzh-rpg/rpg_e2vid

## Overview

This repository is an inference and data-conversion stack for reconstructing intensity images from event-camera streams using a pretrained PyTorch model. The code is organized around a single main pipeline, with several utility scripts that adapt event data to and from ROS and image folders.

#### Overall architecture

The repository has four main subsystems:

1. **Reconstruction entrypoint and pipeline**
   - `run_reconstruction.py`
   - `image_reconstructor.py`

2. **Neural network model**
   - `model/`
   - `base/`

3. **Configuration and shared utilities**
   - `options/`
   - `utils/`

4. **Format conversion and ROS interoperability scripts**
   - `scripts/`

The project is primarily an **inference-time application**, not a training framework. The pretrained weights live under `pretrained/`, and event datasets are expected under `data/`.

#### Main data flow

The central flow is:

1. **Event data input**
   - Events come from a ZIP file or text-based event file format used by the repo.
   - Optional ROS bag workflows convert ROS event topics into this internal format first.

2. **Event windowing and preprocessing**
   - Events are grouped into windows, either by:
     - a fixed number of events, or
     - a fixed time duration.
   - The resulting event window is converted into the tensor format expected by the neural network.
   - Optional preprocessing includes normalization and hot-pixel suppression.

3. **Model inference**
   - The event tensor is passed through a pretrained recurrent reconstruction network.
   - The network produces a reconstructed intensity image for each window.
   - The model can maintain state across windows unless recurrent behavior is disabled.

4. **Post-processing and visualization**
   - Output images are tone-mapped, optionally with automatic HDR scaling.
   - Reconstructions can be displayed live in an OpenCV window.
   - They can also be saved to disk.

5. **Export and interoperability**
   - Reconstructions can be converted back into a ROS bag or embedded into an existing rosbag.
   - A resampling script can turn reconstruction folders into a fixed-rate image sequence for video export.

#### Core reconstruction subsystem

##### `run_reconstruction.py`
This is the top-level executable for the reconstruction pipeline. It is the main user-facing entrypoint described in the README.

Its responsibilities are likely:
- parsing inference options,
- loading the pretrained checkpoint,
- opening the input event source,
- driving the reconstruction loop,
- saving and/or displaying outputs.

##### `image_reconstructor.py`
This is the likely orchestrator class for the actual inference loop. It probably:
- receives batches/windows of events,
- converts them into tensors,
- manages recurrent state,
- invokes the model,
- applies output normalization/tone mapping,
- handles preview rendering and disk output.

This file is the architectural center of the repository: it sits between event I/O and the neural network.

#### Model subsystem

##### `model/`
This folder contains the PyTorch architecture used for event-to-image reconstruction.

Likely structure:
- `model/model.py` — the main model definition and top-level network wrapper.
- `model/unet.py` — the U-Net-style encoder/decoder backbone.
- `model/submodules.py` — reusable layers and blocks used by the network.
- `model/__init__.py` — package export surface.

The README and file names indicate a **U-Net-based recurrent reconstruction model**. The recurrent connection is important because the CLI exposes a `--no-recurrent` option, implying that the model or its wrapper maintains state across successive event windows.

##### `base/`
- `base/base_model.py` likely provides a shared abstract base class or checkpoint-loading conventions inherited from the model implementation.
- This suggests the repo follows a lightweight framework pattern where the concrete model extends a generic PyTorch model wrapper.

#### Configuration subsystem

##### `options/`
- `options/inference_options.py` is the command-line interface definition for reconstruction-time parameters.

This is where the runtime behavior is configured:
- event windowing strategy,
- HDR normalization,
- color reconstruction mode,
- output path and dataset naming,
- display controls,
- normalization and recurrent-state toggles,
- hot-pixel filtering input.

Architecturally, this module is the boundary between human/operator configuration and the reconstruction engine.

#### Utility subsystem

##### `utils/`
These modules provide the supporting infrastructure around the main pipeline:

- `utils/event_readers.py`
  - Event source readers and parsers.
  - Handles file-based event input formats used by the reconstruction pipeline.

- `utils/inference_utils.py`
  - Inference-time helpers for preparing inputs, postprocessing outputs, and possibly managing batching/window logic.

- `utils/loading_utils.py`
  - Model and checkpoint loading helpers.
  - Likely responsible for loading pretrained weights from `.pth.tar` files.

- `utils/path_utils.py`
  - Path and dataset output-folder management.

- `utils/timers.py`
  - Runtime profiling / timing utilities.

- `utils/util.py`
  - Miscellaneous shared functions.

These utilities keep the main reconstruction path thin and isolate file-format and checkpoint concerns from the core model code.

#### ROS and dataset conversion scripts

The `scripts/` folder provides bridges between the internal reconstruction format and external robotics tooling.

##### `scripts/extract_events_from_rosbag.py`
- Reads event streams from a ROS bag.
- Extracts the configured event topic into the repo’s event file/ZIP format.
- This is the import path from ROS into the reconstruction pipeline.

##### `scripts/image_folder_to_rosbag.py`
- Takes a folder of reconstructed images and writes them into a ROS bag.
- Useful for downstream robotics tools expecting image topics.

##### `scripts/embed_reconstructed_images_in_rosbag.py`
- Appends reconstructed images to an existing rosbag.
- Acts as a postprocessing/integration step for experiments.

##### `scripts/resample_reconstructions.py`
- Resamples a reconstruction folder into a fixed-rate image sequence.
- This is a bridge to conventional video workflows.

These scripts are intentionally separate from the main PyTorch inference environment. The README notes that ROS does not need to be installed to run reconstruction itself, only for the ROS-specific scripts.

#### External dependencies that matter architecturally

The runtime is built around:
- **PyTorch** for the neural network and inference,
- **NumPy / Pandas** for data manipulation,
- **OpenCV** for visualization and image I/O,
- **ROS / rosbag** for interoperability scripts.

This matters because the project is split between:
- a pure Python/PyTorch inference environment, and
- optional ROS tooling that lives at the edges.

#### Important boundaries

##### 1. Inference boundary
The repo does not appear to include training pipelines. The key boundary is between:
- pretrained model loading,
- event-window preprocessing,
- model inference,
- image postprocessing.

##### 2. Event format boundary
The core reconstruction code consumes a repository-specific event representation, while ROS conversion scripts adapt external datasets into that format.

##### 3. Output boundary
The reconstructed images can be:
- visualized live,
- written to disk,
- resampled into video-friendly sequences,
- embedded back into ROS bags.

##### 4. Statefulness boundary
The model can operate with or without recurrent state. This means the reconstruction loop must track temporal context across windows unless explicitly disabled.

#### How to draw the architecture

A useful diagram would place the following in sequence:

- **Input sources**
  - ZIP/text event datasets
  - ROS bag event topics

- **Preprocessing / windowing**
  - event readers
  - normalization
  - hot-pixel filtering
  - fixed-event or fixed-duration slicing

- **Reconstruction engine**
  - `image_reconstructor.py`
  - checkpoint loading utilities
  - recurrent state management

- **Neural network**
  - `model/model.py`
  - U-Net backbone in `model/unet.py`
  - supporting blocks in `model/submodules.py`

- **Outputs**
  - saved images
  - OpenCV preview
  - resampled frames/video
  - ROS bag export/embedding

- **Configuration**
  - CLI options in `options/inference_options.py`

This repo is best understood as a focused event-to-video inference system with a small set of adapter scripts around it, rather than as a general application platform.

Main entry points: [`image_reconstructor.py`](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/image_reconstructor.py) (Reconstructor), [`model/model.py`](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/model/model.py) (Reconstruction net), [`run_reconstruction.py`](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/run_reconstruction.py) (Run recon), [`options/inference_options.py`](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/options/inference_options.py) (Inference opts)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph "Reconstruction pipeline"
  node_run_reconstruction["Run recon<br/>entrypoint"]
  node_image_reconstructor["Reconstructor<br/>orchestrator"]
end

subgraph "Neural model"
  node_model_pkg["Model pkg<br/>[__init__.py]"]
  node_model_main["Reconstruction net<br/>network wrapper<br/>[model.py]"]
  node_unet["U-Net backbone<br/>encoder-decoder<br/>[unet.py]"]
  node_submodules["Blocks<br/>layers<br/>[submodules.py]"]
  node_base_model["Base model<br/>pytorch base<br/>[base_model.py]"]
end

subgraph "Config and support"
  node_inference_options["Inference opts<br/>cli config"]
  node_event_readers["Event readers<br/>io parser<br/>[event_readers.py]"]
  node_inference_utils["Inference utils<br/>pre/postprocess<br/>[inference_utils.py]"]
  node_loading_utils["Load weights<br/>checkpoint loader<br/>[loading_utils.py]"]
  node_path_utils["Path utils<br/>paths<br/>[path_utils.py]"]
  node_timers["Timers<br/>profiling<br/>[timers.py]"]
  node_util["Shared util<br/>misc helpers<br/>[util.py]"]
end

subgraph "Conversion scripts"
  node_extract_rosbag["Extract events<br/>ros import"]
  node_image_to_rosbag["Images to bag<br/>ros export"]
  node_embed_rosbag["Embed images<br/>ros postprocess"]
  node_resample["Resample recon<br/>frame export"]
end

node_run_reconstruction -->|"reads CLI"| node_inference_options
node_run_reconstruction -->|"starts"| node_image_reconstructor
node_image_reconstructor -->|"loads events"| node_event_readers
node_image_reconstructor -->|"pre/postprocess"| node_inference_utils
node_image_reconstructor -->|"loads checkpoint"| node_loading_utils
node_image_reconstructor -->|"writes outputs"| node_path_utils
node_image_reconstructor -->|"profiles"| node_timers
node_image_reconstructor -->|"invokes"| node_model_main
node_model_main -->|"extends"| node_base_model
node_model_main -->|"builds on"| node_unet
node_model_main -->|"uses blocks"| node_submodules
node_model_pkg -->|"exports"| node_model_main
node_extract_rosbag -->|"produces events"| node_event_readers
node_image_to_rosbag -.->|"shares export format"| node_embed_rosbag
node_resample -.->|"preps frames"| node_image_to_rosbag
node_inference_options -->|"configures"| node_image_reconstructor
node_util -.->|"supports"| node_inference_utils

click node_run_reconstruction "https://github.com/uzh-rpg/rpg_e2vid/blob/master/run_reconstruction.py"
click node_image_reconstructor "https://github.com/uzh-rpg/rpg_e2vid/blob/master/image_reconstructor.py"
click node_inference_options "https://github.com/uzh-rpg/rpg_e2vid/blob/master/options/inference_options.py"
click node_event_readers "https://github.com/uzh-rpg/rpg_e2vid/blob/master/utils/event_readers.py"
click node_inference_utils "https://github.com/uzh-rpg/rpg_e2vid/blob/master/utils/inference_utils.py"
click node_loading_utils "https://github.com/uzh-rpg/rpg_e2vid/blob/master/utils/loading_utils.py"
click node_path_utils "https://github.com/uzh-rpg/rpg_e2vid/blob/master/utils/path_utils.py"
click node_timers "https://github.com/uzh-rpg/rpg_e2vid/blob/master/utils/timers.py"
click node_util "https://github.com/uzh-rpg/rpg_e2vid/blob/master/utils/util.py"
click node_model_pkg "https://github.com/uzh-rpg/rpg_e2vid/blob/master/model/__init__.py"
click node_model_main "https://github.com/uzh-rpg/rpg_e2vid/blob/master/model/model.py"
click node_unet "https://github.com/uzh-rpg/rpg_e2vid/blob/master/model/unet.py"
click node_submodules "https://github.com/uzh-rpg/rpg_e2vid/blob/master/model/submodules.py"
click node_base_model "https://github.com/uzh-rpg/rpg_e2vid/blob/master/base/base_model.py"
click node_extract_rosbag "https://github.com/uzh-rpg/rpg_e2vid/blob/master/scripts/extract_events_from_rosbag.py"
click node_image_to_rosbag "https://github.com/uzh-rpg/rpg_e2vid/blob/master/scripts/image_folder_to_rosbag.py"
click node_embed_rosbag "https://github.com/uzh-rpg/rpg_e2vid/blob/master/scripts/embed_reconstructed_images_in_rosbag.py"
click node_resample "https://github.com/uzh-rpg/rpg_e2vid/blob/master/scripts/resample_reconstructions.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_run_reconstruction,node_image_reconstructor toneBlue
class node_model_pkg,node_model_main,node_unet,node_submodules,node_base_model toneAmber
class node_inference_options,node_event_readers,node_inference_utils,node_loading_utils,node_path_utils,node_timers,node_util toneMint
class node_extract_rosbag,node_image_to_rosbag,node_embed_rosbag,node_resample toneRose
```

## Components

### Reconstruction pipeline

Top-level inference entrypoint and the reconstruction loop.

- [Run recon](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/run_reconstruction.py): entrypoint · Top-level executable that launches inference and output handling. · `run_reconstruction.py`
- [Reconstructor](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/image_reconstructor.py): orchestrator · Drives event windowing, preprocessing, inference, and output rendering. · `image_reconstructor.py`

### Neural model

PyTorch network and reusable building blocks.

- [Model pkg](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/model/__init__.py): package · Package surface for the reconstruction network implementation. · `model/__init__.py`
- [Reconstruction net](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/model/model.py): network wrapper · Top-level PyTorch model for event-to-image reconstruction. · `model/model.py`
- [U-Net backbone](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/model/unet.py): encoder-decoder · Core U-Net-style feature extractor and decoder backbone. · `model/unet.py`
- [Blocks](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/model/submodules.py): layers · Reusable network layers and building blocks. · `model/submodules.py`
- [Base model](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/base/base_model.py): pytorch base · Shared model base class and common loading conventions. · `base/base_model.py`

### Config and support

CLI options and shared utilities used by inference.

- [Inference opts](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/options/inference_options.py): cli config · Defines reconstruction-time command-line arguments and runtime flags. · `options/inference_options.py`
- [Event readers](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/utils/event_readers.py): io parser · Reads repository event formats and feeds the reconstruction pipeline. · `utils/event_readers.py`
- [Inference utils](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/utils/inference_utils.py): pre/postprocess · Helpers for window preparation, normalization, and output handling. · `utils/inference_utils.py`
- [Load weights](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/utils/loading_utils.py): checkpoint loader · Loads pretrained model checkpoints and restores inference state. · `utils/loading_utils.py`
- [Path utils](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/utils/path_utils.py): paths · Manages dataset, experiment, and output paths. · `utils/path_utils.py`
- [Timers](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/utils/timers.py): profiling · Runtime timing and profiling helpers. · `utils/timers.py`
- [Shared util](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/utils/util.py): misc helpers · General-purpose helper functions used across the repo. · `utils/util.py`

### Conversion scripts

ROS and image-sequence interoperability tools.

- [Extract events](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/scripts/extract_events_from_rosbag.py): ros import · Converts ROS bag event topics into the repo's internal event format. · `scripts/extract_events_from_rosbag.py`
- [Images to bag](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/scripts/image_folder_to_rosbag.py): ros export · Writes reconstructed image folders into a ROS bag. · `scripts/image_folder_to_rosbag.py`
- [Embed images](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/scripts/embed_reconstructed_images_in_rosbag.py): ros postprocess · Appends reconstructed images into an existing ROS bag. · `scripts/embed_reconstructed_images_in_rosbag.py`
- [Resample recon](https://github.com/uzh-rpg/rpg_e2vid/blob/HEAD/scripts/resample_reconstructions.py): frame export · Resamples reconstruction folders into fixed-rate image sequences. · `scripts/resample_reconstructions.py`

## Connections

- Run recon → Inference opts: reads CLI · Command-line arguments configure the reconstruction run.
- Run recon → Reconstructor: starts · Entrypoint creates and runs the reconstruction engine.
- Reconstructor → Event readers: loads events · Consumes repository event formats from input sources.
- Reconstructor → Inference utils: pre/postprocess · Uses helpers for event window preparation and output processing.
- Reconstructor → Load weights: loads checkpoint · Restores pretrained weights for inference.
- Reconstructor → Path utils: writes outputs · Resolves output directories and dataset paths.
- Reconstructor → Timers: profiles · Measures runtime stages of reconstruction.
- Reconstructor → Reconstruction net: invokes · Feeds event tensors into the reconstruction network.
- Reconstruction net → Base model: extends · Uses shared base model conventions.
- Reconstruction net → U-Net backbone: builds on · Wraps the U-Net-style backbone.
- Reconstruction net → Blocks: uses blocks · Composes reusable network layers.
- Model pkg → Reconstruction net: exports · Package exposes the main model implementation.
- Extract events → Event readers: produces events · Converts ROS event topics into the internal event representation.
- Images to bag → Embed images: shares export format · Both scripts target ROS bag image integration workflows.
- Resample recon → Images to bag: preps frames · Resampled reconstructions can feed image-based export workflows.
- Inference opts → Reconstructor: configures · Options control windowing, scaling, display, and recurrent behavior.
- Shared util → Inference utils: supports · General helpers underpin inference-time processing.
