How uzh-rpg/rpg_e2vid works
This repository is an inference and data-conversion stack for reconstructing intensity images from event-camera streams using a pretrained PyTorch model. The code is organized around a single main pipeline, with several utility scripts that adapt event data to and from ROS and image folders.
Overall architecture
The repository has four main subsystems:
1. Reconstruction entrypoint and pipeline
- run_reconstruction.py
- image_reconstructor.py
2. Neural network model
- model/
- base/
3. Configuration and shared utilities
- options/
- utils/
4. Format conversion and ROS interoperability scripts
- scripts/
The project is primarily an inference-time application, not a training framework. The pretrained weights live under pretrained/, and event datasets are expected under data/.
Main data flow
The central flow is:
1. Event data input
- Events come from a ZIP file or text-based event file format used by the repo.
- Optional ROS bag workflows convert ROS event topics into this internal format first.
2. Event windowing and preprocessing
- Events are grouped into windows, either by:
- a fixed number of events, or
- a fixed time duration.
- The resulting event window is converted into the tensor format expected by the neural network.
- Optional preprocessing includes normalization and hot-pixel suppression.
3. Model inference
- The event tensor is passed through a pretrained recurrent reconstruction network.
- The network produces a reconstructed intensity image for each window.
- The model can maintain state across windows unless recurrent behavior is disabled.
4. Post-processing and visualization
- Output images are tone-mapped, optionally with automatic HDR scaling.
- Reconstructions can be displayed live in an OpenCV window.
- They can also be saved to disk.
5. Export and interoperability
- Reconstructions can be converted back into a ROS bag or embedded into an existing rosbag.
- A resampling script can turn reconstruction folders into a fixed-rate image sequence for video export.
Core reconstruction subsystem
run_reconstruction.py
This is the top-level executable for the reconstruction pipeline. It is the main user-facing entrypoint described in the README.
Its responsibilities are likely:
• parsing inference options,
• loading the pretrained checkpoint,
• opening the input event source,
• driving the reconstruction loop,
• saving and/or displaying outputs.
image_reconstructor.py
This is the likely orchestrator class for the actual inference loop. It probably:
• receives batches/windows of events,
• converts them into tensors,
• manages recurrent state,
• invokes the model,
• applies output normalization/tone mapping,
• handles preview rendering and disk output.
This file is the architectural center of the repository: it sits between event I/O and the neural network.
Model subsystem
model/
This folder contains the PyTorch architecture used for event-to-image reconstruction.
Likely structure:
• model/model.py — the main model definition and top-level network wrapper.
• model/unet.py — the U-Net-style encoder/decoder backbone.
• model/submodules.py — reusable layers and blocks used by the network.
• model/__init__.py — package export surface.
The README and file names indicate a U-Net-based recurrent reconstruction model. The recurrent connection is important because the CLI exposes a --no-recurrent option, implying that the model or its wrapper maintains state across successive event windows.
base/
• base/base_model.py likely provides a shared abstract base class or checkpoint-loading conventions inherited from the model implementation.
• This suggests the repo follows a lightweight framework pattern where the concrete model extends a generic PyTorch model wrapper.
Configuration subsystem
options/
• options/inference_options.py is the command-line interface definition for reconstruction-time parameters.
This is where the runtime behavior is configured:
• event windowing strategy,
• HDR normalization,
• color reconstruction mode,
• output path and dataset naming,
• display controls,
• normalization and recurrent-state toggles,
• hot-pixel filtering input.
Architecturally, this module is the boundary between human/operator configuration and the reconstruction engine.
Utility subsystem
utils/
These modules provide the supporting infrastructure around the main pipeline:
• utils/event_readers.py
- Event source readers and parsers.
- Handles file-based event input formats used by the reconstruction pipeline.
• utils/inference_utils.py
- Inference-time helpers for preparing inputs, postprocessing outputs, and possibly managing batching/window logic.
• utils/loading_utils.py
- Model and checkpoint loading helpers.
- Likely responsible for loading pretrained weights from .pth.tar files.
• utils/path_utils.py
- Path and dataset output-folder management.
• utils/timers.py
- Runtime profiling / timing utilities.
• utils/util.py
- Miscellaneous shared functions.
These utilities keep the main reconstruction path thin and isolate file-format and checkpoint concerns from the core model code.
ROS and dataset conversion scripts
The scripts/ folder provides bridges between the internal reconstruction format and external robotics tooling.
scripts/extract_events_from_rosbag.py
• Reads event streams from a ROS bag.
• Extracts the configured event topic into the repo’s event file/ZIP format.
• This is the import path from ROS into the reconstruction pipeline.
scripts/image_folder_to_rosbag.py
• Takes a folder of reconstructed images and writes them into a ROS bag.
• Useful for downstream robotics tools expecting image topics.
scripts/embed_reconstructed_images_in_rosbag.py
• Appends reconstructed images to an existing rosbag.
• Acts as a postprocessing/integration step for experiments.
scripts/resample_reconstructions.py
• Resamples a reconstruction folder into a fixed-rate image sequence.
• This is a bridge to conventional video workflows.
These scripts are intentionally separate from the main PyTorch inference environment. The README notes that ROS does not need to be installed to run reconstruction itself, only for the ROS-specific scripts.
External dependencies that matter architecturally
The runtime is built around:
• PyTorch for the neural network and inference,
• NumPy / Pandas for data manipulation,
• OpenCV for visualization and image I/O,
• ROS / rosbag for interoperability scripts.
This matters because the project is split between:
• a pure Python/PyTorch inference environment, and
• optional ROS tooling that lives at the edges.
Important boundaries
1. Inference boundary
The repo does not appear to include training pipelines. The key boundary is between:
• pretrained model loading,
• event-window preprocessing,
• model inference,
• image postprocessing.
2. Event format boundary
The core reconstruction code consumes a repository-specific event representation, while ROS conversion scripts adapt external datasets into that format.
3. Output boundary
The reconstructed images can be:
• visualized live,
• written to disk,
• resampled into video-friendly sequences,
• embedded back into ROS bags.
4. Statefulness boundary
The model can operate with or without recurrent state. This means the reconstruction loop must track temporal context across windows unless explicitly disabled.
How to draw the architecture
A useful diagram would place the following in sequence:
• Input sources
- ZIP/text event datasets
- ROS bag event topics
• Preprocessing / windowing
- event readers
- normalization
- hot-pixel filtering
- fixed-event or fixed-duration slicing
• Reconstruction engine
- image_reconstructor.py
- checkpoint loading utilities
- recurrent state management
• Neural network
- model/model.py
- U-Net backbone in model/unet.py
- supporting blocks in model/submodules.py
• Outputs
- saved images
- OpenCV preview
- resampled frames/video
- ROS bag export/embedding
• Configuration
- CLI options in options/inference_options.py
This repo is best understood as a focused event-to-video inference system with a small set of adapter scripts around it, rather than as a general application platform.
Main entry points: image_reconstructor.py Reconstructor · model.py Reconstruction net · run_reconstruction.py Run recon · inference_options.py Inference opts
Components 18
Reconstruction pipeline
Top-level inference entrypoint and the reconstruction loop.
- Run recon · Top-level executable that launches inference and output handling.
run_reconstruction.py - Reconstructor · Drives event windowing, preprocessing, inference, and output rendering.
image_reconstructor.py
Neural model
PyTorch network and reusable building blocks.
- Model pkg · Package surface for the reconstruction network implementation.
model/__init__.py - Reconstruction net · Top-level PyTorch model for event-to-image reconstruction.
model/model.py - U-Net backbone · Core U-Net-style feature extractor and decoder backbone.
model/unet.py - Blocks · Reusable network layers and building blocks.
model/submodules.py - Base model · Shared model base class and common loading conventions.
base/base_model.py
Config and support
CLI options and shared utilities used by inference.
- Inference opts · Defines reconstruction-time command-line arguments and runtime flags.
options/inference_options.py - Event readers · Reads repository event formats and feeds the reconstruction pipeline.
utils/event_readers.py - Inference utils · Helpers for window preparation, normalization, and output handling.
utils/inference_utils.py - Load weights · Loads pretrained model checkpoints and restores inference state.
utils/loading_utils.py - Path utils · Manages dataset, experiment, and output paths.
utils/path_utils.py - Timers · Runtime timing and profiling helpers.
utils/timers.py - Shared util · General-purpose helper functions used across the repo.
utils/util.py
Conversion scripts
ROS and image-sequence interoperability tools.
- Extract events · Converts ROS bag event topics into the repo's internal event format.
scripts/extract_events_from_rosbag.py - Images to bag · Writes reconstructed image folders into a ROS bag.
scripts/image_folder_to_rosbag.py - Embed images · Appends reconstructed images into an existing ROS bag.
scripts/embed_reconstructed_images_in_rosbag.py - Resample recon · Resamples reconstruction folders into fixed-rate image sequences.
scripts/resample_reconstructions.py
Connections 17
- Run recon → Inference opts: reads CLI · Command-line arguments configure the reconstruction run.
- Run recon → Reconstructor: starts · Entrypoint creates and runs the reconstruction engine.
- Reconstructor → Event readers: loads events · Consumes repository event formats from input sources.
- Reconstructor → Inference utils: pre/postprocess · Uses helpers for event window preparation and output processing.
- Reconstructor → Load weights: loads checkpoint · Restores pretrained weights for inference.
- Reconstructor → Path utils: writes outputs · Resolves output directories and dataset paths.
- Reconstructor → Timers: profiles · Measures runtime stages of reconstruction.
- Reconstructor → Reconstruction net: invokes · Feeds event tensors into the reconstruction network.
- Reconstruction net → Base model: extends · Uses shared base model conventions.
- Reconstruction net → U-Net backbone: builds on · Wraps the U-Net-style backbone.
- Reconstruction net → Blocks: uses blocks · Composes reusable network layers.
- Model pkg → Reconstruction net: exports · Package exposes the main model implementation.
- Extract events → Event readers: produces events · Converts ROS event topics into the internal event representation.
- Images to bag → Embed images: shares export format · Both scripts target ROS bag image integration workflows.
- Resample recon → Images to bag: preps frames · Resampled reconstructions can feed image-based export workflows.
- Inference opts → Reconstructor: configures · Options control windowing, scaling, display, and recurrent behavior.
- Shared util → Inference utils: supports · General helpers underpin inference-time processing.