How neuroailab/opt_cwm works
Project Type and Purpose
• This project is a research-driven, self-supervised deep learning system for optical flow prediction. It leverages the Counterfactual World Modeling (CWM) paradigm and involves models such as the Flow Predictor and full Opt-CWM model.
• It entails components from data ingestion and pre-processing (e.g., handling Kinetics-400 and TAP-Vid datasets), model training (with self-supervised and distillation modes), evaluation, and demonstration.
File Structure
• Top-Level Directories:
• configs: Contains YAML configuration files for both training and evaluation.
• data: Contains submodules for handling different datasets (kinetics and tapvid) along with utilities for data processing (e.g., downloading data and preparing file lists).
• external: Includes external dependencies, notably a custom version of SEA-RAFT (for distillation and distilled model aspects). This directory shows components such as configuration files, core modules (augmentor, extractor, layer, loss, etc.) and utilities that are integrated into the system.
• models: Contains various model architectures (base_cwm, flow, opt_cwm, two_stream_cwm), each with specialized modules (e.g., masking generators, flow prediction utilities, transformer modules, etc.). This is the heart of the system.
• scripts: Provides shell scripts to easily run training, evaluation (including TAP-Vid evaluation), demo, and RAFT-based distillation.
• utils: Holds common utility functions (e.g., for logging, optimization, options parsing) used across the project.
• demo and README: Provide demonstration examples and documentation.
• The file structure clearly indicates a modular design with separation between data, models, configurations, external code, and execution scripts.
README for Additional Insights
• The README explains key usage scenarios like:
• Running pre-trained models (e.g., Flow Predictor demo via demo.py and run_demo.sh).
• Pre-training on Kinetics-400 and evaluation on TAP-Vid benchmarks.
• Configuration management via YAML files and CLI overrides.
• Distillation using the SEA-RAFT variant.
• It emphasizes the flow from data ingestion, through configuration-driven model instantiation (using builder.get_flow_predictor or builder.get_opt_cwm), and finally, evaluation using provided scripts.
a. Main Components:
• Data Module:
• Kinetics dataset (data/kinetics): Downloading, preparing file lists, and pre-processing.
• TAP-Vid dataset (data/tapvid): Evaluation and processing.
• Configuration Management:
• YAML files in configs (train_cfg.yaml, eval_cfg.yaml, raft_eval_cfg.yaml) that control model parameters.
• Model Building and Inference:
• Core models in models/ (split into base_cwm, flow, opt_cwm, two_stream_cwm).
• Builder module (models/builder.py) for instantiation of model components.
• External SEA-RAFT integration (external/sea_raft): Provides additional modules for distillation and supplementary functionality.
• Utility Components:
• Common utilities (utils/ and parts of external modules).
• Execution Scripts:
• Scripts for training (run_train.sh), evaluation (run_tapvid_eval.sh, run_raft.sh), and demo (run_demo.sh).
• Demonstration and Visualization:
• demo.py and demo assets to show model outputs (e.g., sample optical flow predictions).
b. Relationships and Data Flow:
• Data flows in from datasets (kinetics for training/pre-training, TAP-Vid for evaluation), undergoes processing via custom dataset utilities.
• The configuration layer (configs/) feeds parameters into the model builder.
• The builder instantiates models from the models/ directory which may call utility functions in utils/ and, in the case of the distilled model, modules from external/sea_raft.
• Training scripts (e.g., run_train.sh) orchestrate data loading, model initialization, and training loops.
• Evaluation scripts (run_tapvid_eval.sh, run_raft.sh) create a pipeline to run inference and measure performance.
• A separate demo component shows practical usage by loading pre-trained models and visualizing output using demo.py.
c. Architectural Patterns and Design Principles:
• Modular design: Clear separation of concerns (data, configuration, models, utilities, external libraries).
• Config-driven behavior: YAML configurations combined with CLI options allowing dynamic alteration of model parameters.
• Reusability and extensibility: Different model variants (base_cwm, two_stream_cwm, opt_cwm) and external model distillation introduce plug-and-play capabilities.
• Self-contained evaluation and demo pipelines (via dedicated scripts) to promote reproducibility.
d. Technologies, Frameworks, and Libraries:
• Python environment (Python 3.10 as per installation instructions).
• Deep learning frameworks (implied through model training and inference scripts—possibly PyTorch, though not explicitly mentioned but common in optical flow research).
• YAML for configuration.
• Shell scripting for orchestrating training and evaluation.
• Likely GPU acceleration for training and inference.
Elements
• Data flow from datasets → Data preprocessing → Model Training/Evaluation.
• Configuration injection from configs → Model Builder → Model instantiation.
• Operation control through scripts connecting to training/evaluation workflows.
• Data storage/processing components.
• Model components and their relationships.
• Utility libraries and external dependencies.
• Notes to indicate how pre-trained models can be loaded and how distillation integrates SEA-RAFT functionalities.