How raphael310/sound-spaces works
Project Type and Purpose
• SoundSpaces is an open-source research platform for audio-visual embodied AI. It provides:
• A realistic acoustic simulator (core “soundspaces” Python package)
• Pre-rendered Room Impulse Responses (RIRs) for Matterport3D and Replica
• APIs and scripts for training/evaluating navigation agents that use both audio and visual sensors
• Reference RL baselines (ss_baselines) implementing PPO/ddppo for tasks like AudioGoal, AV-WAN, SAVI
High-Level Architecture Components
a. Core Simulator
• soundspaces package
• continuous_simulator.py: real-time acoustic + visual rendering
• datasets/…_dataset.py: dataset loaders for AudioNav and SemanticAudioNav
• tasks/*.py: task definitions (observations, action spaces, rewards)
b. Pre-Rendered Data
• res/: sample audio files, material maps
• PanoIR/: scripts for panorama IR rendering
• Precomputed RIR directories (external download, via scripts/download_data.py)
c. Config Layer
• configs/: YAML files defining experiments across datasets (mp3d, replica), tasks (av_nav, av_wan, semantic)
• ss_baselines/*/config: baseline-specific experiment definitions
d. Baselines
• ss_baselines/av_nav, av_wan, savi
• run.py entrypoints orchestrate training/eval
• models/: CNNs, RNN encoders, mapper, planner
• ddppo/ppo trainers & algorithms
e. Orchestration Scripts
• scripts/: data-prep, metadata generation, interactive_demo, binauralizer, cache_observations
• examples/: minimal example, Jupyter tutorial
f. External Dependencies
• Habitat API (for scene loading and agent interface)
• PyTorch (RL training, model definitions)
• Docker (Dockerfile, .dockerignore)
Relationships and Data Flows
1. Configuration (YAML)
→ feed to run.py (or scripts)
2. run.py
→ instantiates a Simulator (continuous_simulator) with specified sensors (RGB, Depth, Spectrogram, AudioGoal)
→ loads scene dataset (Matterport3D/Replica) and RIRs
3. Simulator
→ on each step returns observations: images + audio signals (convolved audio)
4. Agent (ss_baselines)
→ receives obs, passes through model (visual_cnn, audio_cnn, state_encoder)
→ outputs action (e.g., move_forward, turn_left)
5. Rollout Storage & Trainer
→ records transitions, computes rewards, updates policy via PPO/ddppo
→ writes checkpoints, logs (TensorBoard)
6. Scripts for Data Processing / Rendering
→ generate metadata (generate_*_metadata.py)
→ convert ambisonic to binaural (ambisonic_to_binaural.py)
→ cache observations for offline use
Architectural Patterns & Principles
• Modular package structure: separation of simulator, tasks, datasets, baselines
• Config-driven design: experiment behavior entirely determined by YAML
• Registry pattern: baseline/common/baseline_registry.py for plugging new agents
• Clear interface boundaries: Simulator ↔ Agent ↔ Trainer