# facebookresearch/2.5d-visual-sound architecture

> Architecture diagram of the GitHub repository facebookresearch/2.5d-visual-sound, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/2.5d-visual-sound
- Repository: https://github.com/facebookresearch/2.5d-visual-sound

## Overview

##### Project type and purpose
- This is a deep-learning research pipeline that takes monaural audio plus video frames as input and learns to predict binaural audio.
- It contains data-preprocessing modules, a training engine (train.py), a demo (demo.py) for inference, and an evaluation script (evaluate.py).

##### File structure
Top-level modules:
- data/ – dataset definitions and data loaders
- models/ – network definitions, loss (criterion), and high-level model API
- options/ – command-line parsing for train, test, and base options
- util/ – utility functions
- train.py – main training orchestration
- demo.py – inference pipeline using trained weights
- evaluate.py – metrics computation on results
- reEncodeAudio.py – optional audio re-encoding for speed

##### Extract additional insights from README
- Uses PyTorch 1.0.0, Python 2.7, CUDA 9.0
- Trains on the FAIR-Play hdf5 splits; supports multi-GPU and TensorBoard logging
- Demo loads visual+audio weights, processes video frames & audio segments, writes output binaural audio

##### Main components
a. Data Sources
– FAIR-Play hdf5 splits (video frames + raw audio)
– (Optional) reEncodeAudio preprocessor
b. Data Loader & Preprocessor
– base_dataset.py, custom_dataset_data_loader.py, data_loader.py
– transforms or augmentations (if any in util/util.py)
c. Model Components
– Audio network (models/networks.py)
– Visual network (models/networks.py)
– Fusion module in audioVisual_model.py (combines audio & visual features)
– Loss computation (models/criterion.py)
d. Training Engine (train.py)
– Options parsing (train_options.py, base_options.py)
– Optimization loop: forward, loss backward, optimizer step
– Checkpointing (save to checkpoints_dir)
– TensorBoard logger
e. Demo & Evaluation
– demo.py: loads audioVisual_model with weights, processes input paths, outputs binaural audio
– evaluate.py: loads outputs + ground truth, computes metrics

##### Relationships and data flow
- Optimizer loop → Checkpoint Store, TensorBoard.
- From Checkpoint Store → Demo & Evaluation modules.

##### Architectural patterns & design principles
- Modular separation: Data, Model, Options, Utilities, Training scripts
- Pipeline pattern: Preprocess → Load → Train → Evaluate → Demo
- Extensibility: options directory allows new modes; models directory holds interchangeable network architectures

##### Detail level
- Options parser feeding configuration into both train.py and demo.py.
- ReEncodeAudio.py as an optional preprocessor upstream of Data Loader.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Preprocessing Phase
    subgraph "Data Preprocessing" 
        direction TB
        ReEncode[/Optional ReEncodeAudio.py/]:::data
    end

    %% Data Sources & Loading Phase
    subgraph "Data Sources & Loader" 
        direction TB
        FAIRPLAY[(FAIR-Play HDF5 Splits)]:::data
        BaseDataset["base_dataset.py"]:::data
        AVDataset["audioVisual_dataset.py"]:::data
        DataLoader["data_loader.py"]:::data
        CustomLoader["custom_dataset_data_loader.py"]:::data
        BaseLoader["base_data_loader.py"]:::data
    end

    %% Model Phase
    subgraph "Model Components" 
        direction TB
        AudioNet["networks.py (Audio & Visual)"]:::model
        Fusion["audioVisual_model.py"]:::model
        Criterion["criterion.py"]:::model
    end

    %% Training Engine Phase
    subgraph "Training Engine" 
        direction TB
        TrainScript[/train.py/]:::train
        BaseOpts["base_options.py"]:::train
        TrainOpts["train_options.py"]:::train
        Util["util/util.py"]:::train
        Checkpoints[(checkpoints_dir)]:::data
        TensorBoard[(TensorBoard Logs)]:::train
    end

    %% Inference & Evaluation Phase
    subgraph "Inference & Evaluation" 
        direction TB
        DemoScript[/demo.py/]:::inference
        EvalScript[/evaluate.py/]:::inference
        TestOpts["test_options.py"]:::inference
        OutputAudio[(Binaural Audio Output)]:::data
        Metrics[(Evaluation Metrics)]:::inference
    end

    %% Connections: Data Flow
    ReEncode -->|re-encoded audio+video| FAIRPLAY
    FAIRPLAY -->|hdf5 batch| BaseLoader
    BaseLoader -->|dataset instance| AVDataset
    AVDataset -->|batch| DataLoader
    CustomLoader -->|batch| DataLoader
    DataLoader -->|batch→| AudioNet
    DataLoader -->|batch→| Fusion

    %% Model internal flow
    AudioNet -->|features→| Fusion
    Fusion -->|predictions→| Criterion
    Criterion -->|loss→| TrainScript

    %% Training loop
    TrainScript -->|config←| BaseOpts
    TrainScript -->|config←| TrainOpts
    TrainScript -->|utils←| Util
    TrainScript -->|optimize→| AudioNet
    TrainScript -->|save→| Checkpoints
    TrainScript -->|log→| TensorBoard

    %% Inference flow
    Checkpoints -->|load weights| DemoScript
    DemoScript -->|process| OutputAudio
    Checkpoints -->|load weights| EvalScript
    EvalScript -->|compare→| Metrics

    %% Options to Demo & Eval
    DemoScript -->|config←| TestOpts
    EvalScript -->|config←| TestOpts

    %% Legend
    subgraph Legend
        direction LR
        D[(Data Store)]:::data
        P[Process/Module]:::train
        S[/Script/]:::inference
    end

    %% Click Events
    click ReEncode "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/reEncodeAudio.py"
    click BaseDataset "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/data/base_dataset.py"
    click AVDataset "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/data/audioVisual_dataset.py"
    click BaseLoader "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/data/base_data_loader.py"
    click CustomLoader "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/data/custom_dataset_data_loader.py"
    click DataLoader "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/data/data_loader.py"
    click AudioNet "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/models/networks.py"
    click Fusion "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/models/audioVisual_model.py"
    click Criterion "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/models/criterion.py"
    click TrainScript "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/train.py"
    click BaseOpts "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/options/base_options.py"
    click TrainOpts "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/options/train_options.py"
    click Util "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/util/util.py"
    click DemoScript "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/demo.py"
    click EvalScript "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/evaluate.py"
    click TestOpts "https://github.com/facebookresearch/2.5d-visual-sound/blob/main/options/test_options.py"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef model fill:#90EE90,stroke:#000,stroke-width:1px
    classDef train fill:#32CD32,stroke:#000,stroke-width:1px
    classDef inference fill:#FFA500,stroke:#000,stroke-width:1px
    classDef legend fill:#f2f2f2,stroke:#000,stroke-width:1px,font-style:italic

    class ReEncode,FAIRPLAY,Checkpoints,OutputAudio,D data
    class BaseDataset,AVDataset,BaseLoader,CustomLoader,DataLoader data
    class AudioNet,Fusion,Criterion model
    class TrainScript,BaseOpts,TrainOpts,TensorBoard,Util train
    class DemoScript,EvalScript,TestOpts inference
```
