# neuroailab/opt_cwm architecture

> Architecture diagram of the GitHub repository neuroailab/opt_cwm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/neuroailab/opt_cwm
- Repository: https://github.com/neuroailab/opt_cwm

## Overview

##### Project Type and Purpose
- This project is a research-driven, self-supervised deep learning system for optical flow prediction. It leverages the Counterfactual World Modeling (CWM) paradigm and involves models such as the Flow Predictor and full Opt-CWM model.
- It entails components from data ingestion and pre-processing (e.g., handling Kinetics-400 and TAP-Vid datasets), model training (with self-supervised and distillation modes), evaluation, and demonstration.

##### File Structure
- Top-Level Directories:
- configs: Contains YAML configuration files for both training and evaluation.
- data: Contains submodules for handling different datasets (kinetics and tapvid) along with utilities for data processing (e.g., downloading data and preparing file lists).
- external: Includes external dependencies, notably a custom version of SEA-RAFT (for distillation and distilled model aspects). This directory shows components such as configuration files, core modules (augmentor, extractor, layer, loss, etc.) and utilities that are integrated into the system.
- models: Contains various model architectures (base_cwm, flow, opt_cwm, two_stream_cwm), each with specialized modules (e.g., masking generators, flow prediction utilities, transformer modules, etc.). This is the heart of the system.
- scripts: Provides shell scripts to easily run training, evaluation (including TAP-Vid evaluation), demo, and RAFT-based distillation.
- utils: Holds common utility functions (e.g., for logging, optimization, options parsing) used across the project.
- demo and README: Provide demonstration examples and documentation.
- The file structure clearly indicates a modular design with separation between data, models, configurations, external code, and execution scripts.

##### README for Additional Insights
- The README explains key usage scenarios like:
- Running pre-trained models (e.g., Flow Predictor demo via demo.py and run_demo.sh).
- Pre-training on Kinetics-400 and evaluation on TAP-Vid benchmarks.
- Configuration management via YAML files and CLI overrides.
- Distillation using the SEA-RAFT variant.
- It emphasizes the flow from data ingestion, through configuration-driven model instantiation (using builder.get_flow_predictor or builder.get_opt_cwm), and finally, evaluation using provided scripts.

a. Main Components:
- Data Module:
- Kinetics dataset (data/kinetics): Downloading, preparing file lists, and pre-processing.
- TAP-Vid dataset (data/tapvid): Evaluation and processing.
- Configuration Management:
- YAML files in configs (train_cfg.yaml, eval_cfg.yaml, raft_eval_cfg.yaml) that control model parameters.
- Model Building and Inference:
- Core models in models/ (split into base_cwm, flow, opt_cwm, two_stream_cwm).
- Builder module (models/builder.py) for instantiation of model components.
- External SEA-RAFT integration (external/sea_raft): Provides additional modules for distillation and supplementary functionality.
- Utility Components:
- Common utilities (utils/ and parts of external modules).
- Execution Scripts:
- Scripts for training (run_train.sh), evaluation (run_tapvid_eval.sh, run_raft.sh), and demo (run_demo.sh).
- Demonstration and Visualization:
- demo.py and demo assets to show model outputs (e.g., sample optical flow predictions).

b. Relationships and Data Flow:
- Data flows in from datasets (kinetics for training/pre-training, TAP-Vid for evaluation), undergoes processing via custom dataset utilities.
- The configuration layer (configs/) feeds parameters into the model builder.
- The builder instantiates models from the models/ directory which may call utility functions in utils/ and, in the case of the distilled model, modules from external/sea_raft.
- Training scripts (e.g., run_train.sh) orchestrate data loading, model initialization, and training loops.
- Evaluation scripts (run_tapvid_eval.sh, run_raft.sh) create a pipeline to run inference and measure performance.
- A separate demo component shows practical usage by loading pre-trained models and visualizing output using demo.py.

c. Architectural Patterns and Design Principles:
- Modular design: Clear separation of concerns (data, configuration, models, utilities, external libraries).
- Config-driven behavior: YAML configurations combined with CLI options allowing dynamic alteration of model parameters.
- Reusability and extensibility: Different model variants (base_cwm, two_stream_cwm, opt_cwm) and external model distillation introduce plug-and-play capabilities.
- Self-contained evaluation and demo pipelines (via dedicated scripts) to promote reproducibility.

d. Technologies, Frameworks, and Libraries:
- Python environment (Python 3.10 as per installation instructions).
- Deep learning frameworks (implied through model training and inference scripts—possibly PyTorch, though not explicitly mentioned but common in optical flow research).
- YAML for configuration.
- Shell scripting for orchestrating training and evaluation.
- Likely GPU acceleration for training and inference.

##### Elements
- Data flow from datasets → Data preprocessing → Model Training/Evaluation.
- Configuration injection from configs → Model Builder → Model instantiation.
- Operation control through scripts connecting to training/evaluation workflows.
- Data storage/processing components.
- Model components and their relationships.
- Utility libraries and external dependencies.
- Notes to indicate how pre-trained models can be loaded and how distillation integrates SEA-RAFT functionalities.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Modules
    subgraph "Data Modules"
        Kinetics["Kinetics Dataset"]:::data
        TAPVid["TAP-Vid Dataset"]:::data
    end

    %% Configuration Management
    subgraph "Configuration Management"
        Config["YAML Configurations"]:::config
    end

    %% Model Building & Inference
    subgraph "Model Building & Inference"
        Builder["Model Builder (builder.py)"]:::model
        BaseCWM["Base CWM & Variants"]:::model
    end

    %% External Module
    subgraph "External Module"
        SEARAFT["SEA-RAFT Integration"]:::external
    end

    %% Utility Components
    subgraph "Utility Components"
        Utils["Common Utilities"]:::utility
    end

    %% Execution Scripts
    subgraph "Execution Scripts"
        Scripts["Training/Evaluation Scripts"]:::script
    end

    %% Demonstration & Visualization
    subgraph "Demonstration & Visualization"
        Demo["Demo Component (demo.py)"]:::demo
    end

    %% Connections: Data Flow and Orchestration
    Kinetics -->|"pre-process"| Scripts
    TAPVid   -->|"pre-process"| Scripts

    Scripts  -->|"orchestrates"| Builder
    Config   -->|"injects-configs"| Builder
    Builder  -->|"instantiates"| BaseCWM
    Builder  -->|"integrates"| SEARAFT
    BaseCWM  -->|"utilizes"| Utils
    Demo     -->|"loads-model"| BaseCWM

    %% Click Events
    click Kinetics "https://github.com/neuroailab/opt_cwm/tree/main/data/kinetics"
    click TAPVid "https://github.com/neuroailab/opt_cwm/tree/main/data/tapvid"
    click Config "https://github.com/neuroailab/opt_cwm/tree/main/configs"
    click Builder "https://github.com/neuroailab/opt_cwm/blob/main/models/builder.py"
    click BaseCWM "https://github.com/neuroailab/opt_cwm/tree/main/models"
    click SEARAFT "https://github.com/neuroailab/opt_cwm/tree/main/external/sea_raft"
    click Utils "https://github.com/neuroailab/opt_cwm/tree/main/utils"
    click Scripts "https://github.com/neuroailab/opt_cwm/tree/main/scripts"
    click Demo "https://github.com/neuroailab/opt_cwm/blob/main/demo.py"

    %% Styles
    classDef data fill:#FFDDCC,stroke:#FFAA88,stroke-width:2px;
    classDef config fill:#FFEEAA,stroke:#DDCC88,stroke-width:2px;
    classDef model fill:#CCFFDD,stroke:#88DD88,stroke-width:2px;
    classDef external fill:#CCCCFF,stroke:#8888DD,stroke-width:2px;
    classDef utility fill:#FFFFCC,stroke:#AAAA55,stroke-width:2px;
    classDef script fill:#FFCCFF,stroke:#DD88DD,stroke-width:2px;
    classDef demo fill:#CCFFFF,stroke:#88CCCC,stroke-width:2px;
```
