How princeton-vl/raft works
Main Components
• Core Optical Flow Modules:
• core/raft.py – the main RAFT model implementation.
• core/corr.py – handles correlation operations between image pairs.
• core/extractor.py – likely used for feature extraction from images.
• core/update.py – responsible for iterative updates in the recurrent structure.
• Data Handling:
• core/datasets.py – manages loading, processing, and linking to external datasets (FlyingChairs, Sintel, etc.).
• Utilities:
• core/utils folder contains helper modules such as augmentor (data augmentation), flow_viz (visualization), frame_utils (frame handling), and generic utility functions.
• Alternate CUDA Extension:
• alt_cuda_corr directory – an optional, efficient alternative for computing correlations using CUDA. This component integrates with the main execution when the “--alternate_corr” flag is used. It includes both C++/CUDA parts (correlation.cpp, correlation_kernel.cu) and a Python setup script.
• Execution Interfaces:
• demo.py – for running inference/demos on a sequence of frames.
• evaluate.py – for evaluating a trained model.
• train.py and training shell scripts (train_standard.sh, train_mixed.sh) – for initiating training.
• Supporting Files:
• download_models.sh – handles downloading pretrained models.
• external dataset directories (suggested in the README for datasets linking).
Out the Relationships and Interactions
• Data Flow:
• Data ingestion and preparation is handled by the datasets module, which loads various optical flow datasets.
• The extracted image data is then passed to the core network (RAFT model) that uses feature extraction (extractor.py) and correlation (corr.py or alt_cuda_corr if enabled) to compute optical flow.
• Training and Evaluation:
• The training scripts (train.py along with train_standard.sh/train_mixed.sh) orchestrate model training; they will invoke the core modules.
• Evaluation and demo scripts (evaluate.py, demo.py) share a similar dependency on the core architecture.
• Utility Integration:
• The utils modules provide supporting functionality for data augmentation, visualization, and miscellaneous operations which are called at various points in data handling, training, and inference.
Architectural Patterns and Technologies
• Modular Design: The clear separation between core algorithm (RAFT and its sub-components), data handling, and utilities simplifies integration and independent evolution of each module.
• Optional CUDA Extension: the design allows toggling between a standard and an alternate (CUDA-accelerated) correlation method depending on the flag.
• Usage of PyTorch: The README shows dependencies on PyTorch for model building and training.
• External Data and Pretrained Models: There’s an integration aspect with public datasets and downloadable pre-trained model states.
• Deployment/Experimentation: Training, testing, and demo modules point towards experimental research code rather than production “microservices.”