# msu-video-group/memfof architecture

> Architecture diagram of the GitHub repository msu-video-group/memfof, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/msu-video-group/memfof
- Repository: https://github.com/msu-video-group/memfof

## Overview

##### Project type and purpose
- MEMFOF is a research-oriented, memory-efficient multi-frame optical flow estimation framework built in PyTorch. It provides:
- Model definition and utilities (core/…)
- Training pipelines with configurable hyperparameters (config/ + scripts/train.py + train.sh)
- Evaluation and submission tools for multiple benchmarks (scripts/evaluate.py, eval.sh, submission.sh)
- Demo interface (demo.py / demo.ipynb)

##### File structure and infer architectural layers
Top-level directories & files:
- config/            ← JSON configs for training, evaluation, dataset splits
- core/              ← PyTorch modules (network layers, dataset loaders, loss, etc.)
- scripts/           ← CLI entry points (train, evaluate, submission)
- demo.*             ← Interactive demo (Notebook & script)
- eval.sh, train.sh, submission.sh ← Cluster orchestration wrappers
- requirements.txt   ← Key dependencies
- README.md          ← High-level overview & instructions

##### High-level architecture components
a. Configuration layer
– JSON files in config/train, config/eval, config/splits
– parser.py reads configs and builds hyperparameter Namespace
b. Data layer
– core/datasets.py locates datasets, applies transforms (core/utils/flow_transforms.py, augmentor.py)
– config/splits defines train/test partitions
c. Model layer
– core/extractor.py, core/corr.py, core/update.py, core/gma.py, core/memfof.py
– Implements multi-frame optical flow network, memory-efficient modules
d. Training orchestration
– scripts/train.py uses PyTorch Lightning (Trainer, callbacks, cluster launch via torchrun or Slurm)
– train.sh / submit.sh wrap resource requests (nodes, GPUs, CPU cores)
– Weights & Biases integration for logging (WandB)
e. Evaluation & submission
– scripts/evaluate.py loads checkpoints, runs inference on benchmarks (Sintel, KITTI, Spring)
– submission.sh generates formatted outputs for leaderboard submission
– eval.sh batch-evaluates across multiple configurations
f. Demo interface
– demo.py / demo.ipynb for running single-video inference and visualization (core/utils/flow_viz.py)

##### Relationships & data flows
1) User invokes a CLI script (train.py, evaluate.py or demo.py) or shell wrapper (train.sh, eval.sh, demo.ipynb).
2) CLI loads a config file via config/parser.py → populates hyperparameters, dataset paths, model checkpoints.
3) Data loaders (core/datasets.py) read images/flow from disk; apply augmentations from core/utils.
4) Model (core/memfof.py) forward pass: feature extractor → correlation → memory update → flow prediction.
5) Training loop (PyTorch Lightning) backpropagates via core/loss.py; logs metrics to WandB.
6) Checkpoints are saved → later consumed by evaluation or submission scripts.
7) Evaluation script loads checkpoint → dataset split → model inference → metrics calculation → formatted output.
8) Demo pipeline runs inference on arbitrary video, outputs visualizations via core/utils/flow_viz.py.

##### Architectural patterns & design principles
- Modular separation: config vs. core code vs. scripts vs. demo
- Pipeline pattern: clear stages (load config → prepare data → define model → train/evaluate/infer)
- PyTorch Lightning for clean Trainer abstraction and distributed scaling
- Configuration‐driven design: JSON files control nearly all hyperparameters and dataset splits
- Extensibility: core modules can be replaced or extended (e.g., swap memory module or feature extractor)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Layer
    subgraph "Configuration Layer"
        direction TB
        Parser["Config Parser"]:::config
        TrainConfs["Training Configs"]:::config
        EvalConfs["Evaluation Configs"]:::config
        SplitConfs["Dataset Split Configs"]:::config
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DataLoader["Dataset Loader"]:::data
        Augmentor["Augmentations"]:::data
        FlowTransforms["Flow Transforms"]:::data
        FrameUtils["Frame Utilities"]:::data
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        Extractor["Feature Extractor"]:::model
        Corr["Correlation Module"]:::model
        Update["Memory Update Module"]:::model
        GMA["GMA Module"]:::model
        MEMFOF["Core MEMFOF Network"]:::model
        LitWrapper["PyTorch Lightning Wrapper"]:::model
        Loss["Loss Functions"]:::model
        Utils["Utility Helpers"]:::model
    end

    %% Training Orchestration
    subgraph "Training Orchestration"
        direction TB
        TrainScript["Training Script"]:::orches
        ClusterWrapper["Cluster Submission Wrapper"]:::orches
        CKpts["Checkpoint Utilities"]:::orches
        Deps["Dependencies"]:::orches
    end

    %% Evaluation & Submission
    subgraph "Evaluation & Submission"
        direction TB
        EvalScript["Evaluation Script"]:::orches
        SubScript["Submission Script"]:::orches
        EvalWrapper["Evaluation Batch Wrapper"]:::orches
        SubWrapper["Submission Wrapper"]:::orches
    end

    %% Demo Interface
    subgraph "Demo Interface"
        direction TB
        DemoScript["Demo Script"]:::orches
        DemoNB["Interactive Notebook"]:::orches
        FlowViz["Flow Visualization Utilities"]:::data
    end

    %% External Services
    subgraph "External Services"
        direction TB
        WandB["Weights & Biases"]:::external
        HF["Hugging Face"]:::external
        Cluster["Slurm/torchrun"]:::external
    end

    %% Data & Control Flow
    TrainScript -->|invokes| Parser
    EvalScript -->|invokes| Parser
    DemoScript -->|invokes| Parser

    Parser -->|loads configs| TrainConfs
    Parser -->|loads configs| EvalConfs
    Parser -->|loads configs| SplitConfs

    Parser -->|provides hyperparams| TrainScript
    Parser -->|provides hyperparams| EvalScript
    Parser -->|provides hyperparams| DemoScript

    TrainConfs --> Parser
    EvalConfs --> Parser
    SplitConfs --> Parser

    TrainScript -->|calls| DataLoader
    DataLoader -->|uses| Augmentor
    DataLoader -->|uses| FlowTransforms
    DataLoader -->|uses| FrameUtils

    DataLoader -->|feeds data| Extractor
    Extractor --> Corr --> Update --> MEMFOF
    MEMFOF --> LitWrapper
    LitWrapper --> Loss

    TrainScript -->|orchestrates Trainer| LitWrapper
    ClusterWrapper -->|launches| TrainScript
    TrainScript -->|logs| WandB
    TrainScript -->|stores| CKpts

    CKpts -->|provides| EvalScript
    CKpts -->|provides| DemoScript

    EvalScript -->|loads| CKpts
    EvalScript -->|runs inference| MEMFOF
    EvalScript -->|outputs metrics| EvalWrapper
    EvalScript -->|formats submission| SubWrapper

    DemoScript -->|runs inference| MEMFOF
    DemoScript -->|visualizes| FlowViz

    ClusterWrapper --> Cluster
    TrainScript --> Cluster

    %% Click Events for component mapping
    click Parser "https://github.com/msu-video-group/memfof/blob/dev/config/parser.py"
    click TrainConfs "https://github.com/msu-video-group/memfof/tree/dev/config/train"
    click EvalConfs "https://github.com/msu-video-group/memfof/tree/dev/config/eval"
    click SplitConfs "https://github.com/msu-video-group/memfof/tree/dev/config/splits"
    click DataLoader "https://github.com/msu-video-group/memfof/blob/dev/core/datasets.py"
    click Augmentor "https://github.com/msu-video-group/memfof/blob/dev/core/utils/augmentor.py"
    click FlowTransforms "https://github.com/msu-video-group/memfof/blob/dev/core/utils/flow_transforms.py"
    click FrameUtils "https://github.com/msu-video-group/memfof/blob/dev/core/utils/frame_utils.py"
    click Extractor "https://github.com/msu-video-group/memfof/blob/dev/core/extractor.py"
    click Corr "https://github.com/msu-video-group/memfof/blob/dev/core/corr.py"
    click Update "https://github.com/msu-video-group/memfof/blob/dev/core/update.py"
    click GMA "https://github.com/msu-video-group/memfof/blob/dev/core/gma.py"
    click MEMFOF "https://github.com/msu-video-group/memfof/blob/dev/core/memfof.py"
    click LitWrapper "https://github.com/msu-video-group/memfof/blob/dev/core/memfof_lit.py"
    click Loss "https://github.com/msu-video-group/memfof/blob/dev/core/loss.py"
    click Utils "https://github.com/msu-video-group/memfof/blob/dev/core/utils/utils.py"
    click TrainScript "https://github.com/msu-video-group/memfof/blob/dev/scripts/train.py"
    click ClusterWrapper "https://github.com/msu-video-group/memfof/blob/dev/train.sh"
    click CKpts "https://github.com/msu-video-group/memfof/blob/dev/scripts/ckpts.py"
    click Deps "https://github.com/msu-video-group/memfof/blob/dev/requirements.txt"
    click EvalScript "https://github.com/msu-video-group/memfof/blob/dev/scripts/evaluate.py"
    click SubScript "https://github.com/msu-video-group/memfof/blob/dev/scripts/submission.py"
    click EvalWrapper "https://github.com/msu-video-group/memfof/blob/dev/eval.sh"
    click SubWrapper "https://github.com/msu-video-group/memfof/blob/dev/submission.sh"
    click DemoScript "https://github.com/msu-video-group/memfof/blob/dev/demo.py"
    click DemoNB "https://github.com/msu-video-group/memfof/blob/dev/demo.ipynb"
    click FlowViz "https://github.com/msu-video-group/memfof/blob/dev/core/utils/flow_viz.py"

    %% Styles
    classDef config fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef data fill:#add8e6,stroke:#333,stroke-width:1px
    classDef model fill:#90ee90,stroke:#333,stroke-width:1px
    classDef orches fill:#ffa500,stroke:#333,stroke-width:1px
    classDef external fill:#dda0dd,stroke:#333,stroke-width:1px
```
