How ispc-lab/tma works
Project Type and Purpose
• This is a research‐oriented deep‐learning codebase implementing the TMA network for event‐based optical flow.
• Key phases: data generation (“data_utils”), data loading & preprocessing (“dataloader”), model definition (“model”), training (“train.py” / “train_split.py”), and evaluation (“evaluate.py”).
File Structure
• Top level:
• data_utils/ → scripts to convert raw DSEC data into .npz/.npy volumes
• dataloader/ → PyTorch Dataset and augmentation classes
• datasets/ → prepared data on disk
• model/ → network modules (backbone, correlation, aggregation, update, TMA head)
• train.py / train_split.py / train.sh → training entrypoints
• evaluate.py → inference and benchmark submission
• utils/ → file I/O utilities
• Notice “train_split.py” uses a smaller “dsec_split” for rapid prototyping, while “train.sh” orchestrates full‐dataset runs.
Gather Insights from README
• Preprocessing follows E-RAFT style (event volumes + flow arrays).
• Data lives under datasets/dsec_full or datasets/dsec_split.
• Training can log to WandB, outputs saved under a checkpoint directory.
• Evaluation script produces submissions for the online benchmark.
Main Components & Relationships
A. Data Preparation Layer
• data_utils/gen_dsec.py → reads raw .npz/.npy, writes processed event volumes and flows
• data_utils/gen_dsec_upload.py → prepares test‐time volumes for benchmark upload
B. Dataset Storage
• datasets/dsec_full/… → trainval/test subfolders with .npz/.npy
• datasets/dsec_split/… → small train/val splits for development
C. Data Loading & Augmentation
• dataloader/dsec_full.py and dataloader/dsec_split.py create PyTorch Datasets
• dataloader/augment.py applies spatial/temporal augmentations
• dataloader/representation.py converts raw events into network inputs
D. Model Definition
• model/backbone.py → feature extractor
• model/corr.py → correlation cost volume
• model/aggregate.py → temporal motion aggregation module
• model/update.py → flow field refinement
• model/TMA.py → overall network assembly
• model/util.py → helpers (tensor ops, warping)
E. Training & Checkpointing
• train.py → full‐dataset training loop
• train_split.py → training on split dataset
• train.sh → shell wrapper for train.py
• utils/file_utils.py → checkpoint I/O, logging
• Optionally uses WandB for loss visualization
F. Evaluation & Submission
• evaluate.py → loads checkpoint, runs inference on test set, writes .npz for benchmark upload