# princeton-vl/raft architecture

> Architecture diagram of the GitHub repository princeton-vl/raft, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/princeton-vl/raft
- Repository: https://github.com/princeton-vl/raft

## Overview

##### Main Components
- Core Optical Flow Modules:
- core/raft.py – the main RAFT model implementation.
- core/corr.py – handles correlation operations between image pairs.
- core/extractor.py – likely used for feature extraction from images.
- core/update.py – responsible for iterative updates in the recurrent structure.
- Data Handling:
- core/datasets.py – manages loading, processing, and linking to external datasets (FlyingChairs, Sintel, etc.).
- Utilities:
- core/utils folder contains helper modules such as augmentor (data augmentation), flow_viz (visualization), frame_utils (frame handling), and generic utility functions.
- Alternate CUDA Extension:
- alt_cuda_corr directory – an optional, efficient alternative for computing correlations using CUDA. This component integrates with the main execution when the “--alternate_corr” flag is used. It includes both C++/CUDA parts (correlation.cpp, correlation_kernel.cu) and a Python setup script.
- Execution Interfaces:
- demo.py – for running inference/demos on a sequence of frames.
- evaluate.py – for evaluating a trained model.
- train.py and training shell scripts (train_standard.sh, train_mixed.sh) – for initiating training.
- Supporting Files:
- download_models.sh – handles downloading pretrained models.
- external dataset directories (suggested in the README for datasets linking).

##### Out the Relationships and Interactions
- Data Flow:
- Data ingestion and preparation is handled by the datasets module, which loads various optical flow datasets.
- The extracted image data is then passed to the core network (RAFT model) that uses feature extraction (extractor.py) and correlation (corr.py or alt_cuda_corr if enabled) to compute optical flow.
- Training and Evaluation:
- The training scripts (train.py along with train_standard.sh/train_mixed.sh) orchestrate model training; they will invoke the core modules.
- Evaluation and demo scripts (evaluate.py, demo.py) share a similar dependency on the core architecture.
- Utility Integration:
- The utils modules provide supporting functionality for data augmentation, visualization, and miscellaneous operations which are called at various points in data handling, training, and inference.

##### Architectural Patterns and Technologies
- Modular Design: The clear separation between core algorithm (RAFT and its sub-components), data handling, and utilities simplifies integration and independent evolution of each module.
- Optional CUDA Extension: the design allows toggling between a standard and an alternate (CUDA-accelerated) correlation method depending on the flag.
- Usage of PyTorch: The README shows dependencies on PyTorch for model building and training.
- External Data and Pretrained Models: There’s an integration aspect with public datasets and downloadable pre-trained model states.
- Deployment/Experimentation: Training, testing, and demo modules point towards experimental research code rather than production “microservices.”

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Ingestion Layer"
        DS["Datasets Management"]:::data
    end

    subgraph "Model Computation Layer"
        EX["Feature Extraction"]:::core
        COR["Correlation Operations"]:::core
        UPD["Iterative Update"]:::core
        RAFT["RAFT Model"]:::core
        ALT["CUDA Extended Correlation"]:::optional
    end

    subgraph "Utilities/Support Layer"
        AUG["Data Augmentation"]:::util
        FV["Optical Flow Visualization"]:::util
        FU["Frame Utilities"]:::util
        UTIL["Utility Functions"]:::util
    end

    subgraph "Execution Interfaces"
        DEMO["Inference/Demo"]:::exec
        EVAL["Evaluation"]:::exec
        TRAIN["Training Execution"]:::exec
    end

    subgraph "Supporting Files"
        DOWN["Download Pretrained Models"]:::support
    end

    %% Data Flow Connections
    DS -->|"data_input"| EX
    EX -->|"features"| RAFT
    COR -->|"correlations"| RAFT
    UPD -->|"iterative_updates"| RAFT
    ALT -->|"alt_corr"| RAFT

    %% Utility Connections
    DS -->|"augmentation"| AUG
    RAFT -->|"visualize"| FV

    %% Execution Connections
    RAFT -->|"inference"| DEMO
    RAFT -->|"evaluation"| EVAL
    RAFT -->|"training"| TRAIN
    TRAIN -->|"downloads"| DOWN

    %% Class Definitions for Colors
    classDef data fill:#F9D5BB,stroke:#333,stroke-width:2px;
    classDef core fill:#AED6F1,stroke:#333,stroke-width:2px;
    classDef util fill:#D5F5E3,stroke:#333,stroke-width:2px;
    classDef exec fill:#FDEBD0,stroke:#333,stroke-width:2px;
    classDef optional fill:#F5B7B1,stroke:#333,stroke-width:2px;
    classDef support fill:#F9E79F,stroke:#333,stroke-width:2px;

    %% Click Events for Component Mapping
    click RAFT "https://github.com/princeton-vl/raft/blob/master/core/raft.py"
    click COR "https://github.com/princeton-vl/raft/blob/master/core/corr.py"
    click EX "https://github.com/princeton-vl/raft/blob/master/core/extractor.py"
    click UPD "https://github.com/princeton-vl/raft/blob/master/core/update.py"
    click DS "https://github.com/princeton-vl/raft/blob/master/core/datasets.py"
    click AUG "https://github.com/princeton-vl/raft/blob/master/core/utils/augmentor.py"
    click FV "https://github.com/princeton-vl/raft/blob/master/core/utils/flow_viz.py"
    click FU "https://github.com/princeton-vl/raft/blob/master/core/utils/frame_utils.py"
    click UTIL "https://github.com/princeton-vl/raft/blob/master/core/utils/utils.py"
    click ALT "https://github.com/princeton-vl/raft/tree/master/alt_cuda_corr"
    click DEMO "https://github.com/princeton-vl/raft/blob/master/demo.py"
    click EVAL "https://github.com/princeton-vl/raft/blob/master/evaluate.py"
    click TRAIN "https://github.com/princeton-vl/raft/blob/master/train.py"
    click DOWN "https://github.com/princeton-vl/raft/blob/master/download_models.sh"
```
