How msu-video-group/memfof works
Project type and purpose
• MEMFOF is a research-oriented, memory-efficient multi-frame optical flow estimation framework built in PyTorch. It provides:
• Model definition and utilities (core/…)
• Training pipelines with configurable hyperparameters (config/ + scripts/train.py + train.sh)
• Evaluation and submission tools for multiple benchmarks (scripts/evaluate.py, eval.sh, submission.sh)
• Demo interface (demo.py / demo.ipynb)
File structure and infer architectural layers
Top-level directories & files:
• config/ ← JSON configs for training, evaluation, dataset splits
• core/ ← PyTorch modules (network layers, dataset loaders, loss, etc.)
• scripts/ ← CLI entry points (train, evaluate, submission)
• demo.* ← Interactive demo (Notebook & script)
• eval.sh, train.sh, submission.sh ← Cluster orchestration wrappers
• requirements.txt ← Key dependencies
• README.md ← High-level overview & instructions
High-level architecture components
a. Configuration layer
– JSON files in config/train, config/eval, config/splits
– parser.py reads configs and builds hyperparameter Namespace
b. Data layer
– core/datasets.py locates datasets, applies transforms (core/utils/flow_transforms.py, augmentor.py)
– config/splits defines train/test partitions
c. Model layer
– core/extractor.py, core/corr.py, core/update.py, core/gma.py, core/memfof.py
– Implements multi-frame optical flow network, memory-efficient modules
d. Training orchestration
– scripts/train.py uses PyTorch Lightning (Trainer, callbacks, cluster launch via torchrun or Slurm)
– train.sh / submit.sh wrap resource requests (nodes, GPUs, CPU cores)
– Weights & Biases integration for logging (WandB)
e. Evaluation & submission
– scripts/evaluate.py loads checkpoints, runs inference on benchmarks (Sintel, KITTI, Spring)
– submission.sh generates formatted outputs for leaderboard submission
– eval.sh batch-evaluates across multiple configurations
f. Demo interface
– demo.py / demo.ipynb for running single-video inference and visualization (core/utils/flow_viz.py)
Relationships & data flows
1) User invokes a CLI script (train.py, evaluate.py or demo.py) or shell wrapper (train.sh, eval.sh, demo.ipynb).
2) CLI loads a config file via config/parser.py → populates hyperparameters, dataset paths, model checkpoints.
3) Data loaders (core/datasets.py) read images/flow from disk; apply augmentations from core/utils.
4) Model (core/memfof.py) forward pass: feature extractor → correlation → memory update → flow prediction.
5) Training loop (PyTorch Lightning) backpropagates via core/loss.py; logs metrics to WandB.
6) Checkpoints are saved → later consumed by evaluation or submission scripts.
7) Evaluation script loads checkpoint → dataset split → model inference → metrics calculation → formatted output.
8) Demo pipeline runs inference on arbitrary video, outputs visualizations via core/utils/flow_viz.py.
Architectural patterns & design principles
• Modular separation: config vs. core code vs. scripts vs. demo
• Pipeline pattern: clear stages (load config → prepare data → define model → train/evaluate/infer)
• PyTorch Lightning for clean Trainer abstraction and distributed scaling
• Configuration‐driven design: JSON files control nearly all hyperparameters and dataset splits
• Extensibility: core modules can be replaced or extended (e.g., swap memory module or feature extractor)