How facebookresearch/2.5d-visual-sound works
Project type and purpose
• This is a deep-learning research pipeline that takes monaural audio plus video frames as input and learns to predict binaural audio.
• It contains data-preprocessing modules, a training engine (train.py), a demo (demo.py) for inference, and an evaluation script (evaluate.py).
File structure
Top-level modules:
• data/ – dataset definitions and data loaders
• models/ – network definitions, loss (criterion), and high-level model API
• options/ – command-line parsing for train, test, and base options
• util/ – utility functions
• train.py – main training orchestration
• demo.py – inference pipeline using trained weights
• evaluate.py – metrics computation on results
• reEncodeAudio.py – optional audio re-encoding for speed
Extract additional insights from README
• Uses PyTorch 1.0.0, Python 2.7, CUDA 9.0
• Trains on the FAIR-Play hdf5 splits; supports multi-GPU and TensorBoard logging
• Demo loads visual+audio weights, processes video frames & audio segments, writes output binaural audio
Main components
a. Data Sources
– FAIR-Play hdf5 splits (video frames + raw audio)
– (Optional) reEncodeAudio preprocessor
b. Data Loader & Preprocessor
– base_dataset.py, custom_dataset_data_loader.py, data_loader.py
– transforms or augmentations (if any in util/util.py)
c. Model Components
– Audio network (models/networks.py)
– Visual network (models/networks.py)
– Fusion module in audioVisual_model.py (combines audio & visual features)
– Loss computation (models/criterion.py)
d. Training Engine (train.py)
– Options parsing (train_options.py, base_options.py)
– Optimization loop: forward, loss backward, optimizer step
– Checkpointing (save to checkpoints_dir)
– TensorBoard logger
e. Demo & Evaluation
– demo.py: loads audioVisual_model with weights, processes input paths, outputs binaural audio
– evaluate.py: loads outputs + ground truth, computes metrics
Relationships and data flow
• Optimizer loop → Checkpoint Store, TensorBoard.
• From Checkpoint Store → Demo & Evaluation modules.
Architectural patterns & design principles
• Modular separation: Data, Model, Options, Utilities, Training scripts
• Pipeline pattern: Preprocess → Load → Train → Evaluate → Demo
• Extensibility: options directory allows new modes; models directory holds interchangeable network architectures
Detail level
• Options parser feeding configuration into both train.py and demo.py.
• ReEncodeAudio.py as an optional preprocessor upstream of Data Loader.