# raphael310/sound-spaces architecture

> Architecture diagram of the GitHub repository raphael310/sound-spaces, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/raphael310/sound-spaces
- Repository: https://github.com/raphael310/sound-spaces

## Overview

##### Project Type and Purpose
- SoundSpaces is an open-source research platform for audio-visual embodied AI. It provides:
- A realistic acoustic simulator (core “soundspaces” Python package)
- Pre-rendered Room Impulse Responses (RIRs) for Matterport3D and Replica
- APIs and scripts for training/evaluating navigation agents that use both audio and visual sensors
- Reference RL baselines (ss_baselines) implementing PPO/ddppo for tasks like AudioGoal, AV-WAN, SAVI

##### High-Level Architecture Components
a. Core Simulator
- soundspaces package
- continuous_simulator.py: real-time acoustic + visual rendering
- datasets/…_dataset.py: dataset loaders for AudioNav and SemanticAudioNav
- tasks/*.py: task definitions (observations, action spaces, rewards)
b. Pre-Rendered Data
- res/: sample audio files, material maps
- PanoIR/: scripts for panorama IR rendering
- Precomputed RIR directories (external download, via scripts/download_data.py)
c. Config Layer
- configs/: YAML files defining experiments across datasets (mp3d, replica), tasks (av_nav, av_wan, semantic)
- ss_baselines/*/config: baseline-specific experiment definitions
d. Baselines
- ss_baselines/av_nav, av_wan, savi
- run.py entrypoints orchestrate training/eval
- models/: CNNs, RNN encoders, mapper, planner
- ddppo/ppo trainers & algorithms
e. Orchestration Scripts
- scripts/: data-prep, metadata generation, interactive_demo, binauralizer, cache_observations
- examples/: minimal example, Jupyter tutorial
f. External Dependencies
- Habitat API (for scene loading and agent interface)
- PyTorch (RL training, model definitions)
- Docker (Dockerfile, .dockerignore)

##### Relationships and Data Flows
1. Configuration (YAML)
→ feed to run.py (or scripts)
2. run.py
→ instantiates a Simulator (continuous_simulator) with specified sensors (RGB, Depth, Spectrogram, AudioGoal)
→ loads scene dataset (Matterport3D/Replica) and RIRs
3. Simulator
→ on each step returns observations: images + audio signals (convolved audio)
4. Agent (ss_baselines)
→ receives obs, passes through model (visual_cnn, audio_cnn, state_encoder)
→ outputs action (e.g., move_forward, turn_left)
5. Rollout Storage & Trainer
→ records transitions, computes rewards, updates policy via PPO/ddppo
→ writes checkpoints, logs (TensorBoard)
6. Scripts for Data Processing / Rendering
→ generate metadata (generate_*_metadata.py)
→ convert ambisonic to binaural (ambisonic_to_binaural.py)
→ cache observations for offline use

##### Architectural Patterns & Principles
- Modular package structure: separation of simulator, tasks, datasets, baselines
- Config-driven design: experiment behavior entirely determined by YAML
- Registry pattern: baseline/common/baseline_registry.py for plugging new agents
- Clear interface boundaries: Simulator ↔ Agent ↔ Trainer

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Config & Orchestration (Orange)
    subgraph "Config & Orchestration"
        direction TB
        CFG["Config Loader (YAML)"]:::orange
        RUN["Runner / Experiment Orchestrator (run.py)"]:::orange
        SCRIPTS["Orchestration Scripts"]:::orange
    end

    CFG -->|"feeds config"| RUN
    subgraph "Scripts"
        direction TB
        DL["download_data.py"]:::orange
        GEN_MP["generate_matterport3d_metadata.py"]:::orange
        GEN_REP["generate_replica_metadata.py"]:::orange
        AMB["ambisonic_to_binaural.py"]:::orange
        CACHE["cache_observations.py"]:::orange
        DEMO["interactive_demo.py"]:::orange
    end
    SCRIPTS --> DL
    SCRIPTS --> GEN_MP
    SCRIPTS --> GEN_REP
    SCRIPTS --> AMB
    SCRIPTS --> CACHE
    SCRIPTS --> DEMO

    %% Data (Grey)
    subgraph "Pre-Rendered Data & Datasets"
        direction TB
        RES["res/ (Material Map, audio samples)"]:::grey
        PANO["PanoIR Rendering Scripts"]:::grey
        DATASETS["Datasets (Matterport3D, Replica & RIRs)"]:::grey
    end
    DL --> DATASETS
    GEN_MP --> DATASETS
    GEN_REP --> DATASETS
    RUN --> DATASETS

    RES -.-> PANO
    PANO --> DATASETS

    %% Core Simulator (Blue)
    subgraph "Core Simulator (soundspaces)"
        direction TB
        SIM_PKG["soundspaces package"]:::blue
        CSIM["continuous_simulator.py"]:::blue
        SIM["simulator.py"]:::blue
        DATA_LOAD["Dataset Loaders"]:::blue
        TASKS["Task Definitions"]:::blue
    end
    RUN -->|"instantiate"| CSIM
    RUN --> SIM
    CSIM -->|"uses"| DATA_LOAD
    CSIM -->|"uses"| TASKS
    DATASETS -->|"loaded by"| DATA_LOAD

    %% Agent / Baselines (Green)
    subgraph "Agent Modules (ss_baselines)"
        direction TB
        subgraph "Common Utilities"
            BASE_REG["baseline_registry.py"]:::green
            BASE_TR["base_trainer.py"]:::green
            ROLL["rollout_storage.py"]:::green
        end
        subgraph "AV-NAV Baseline"
            direction TB
            AVNAV_RUN["av_nav/run.py"]:::green
            AVNAV_MOD["av_nav/models"]:::green
            AVNAV_PPO["av_nav/ppo & ddppo"]:::green
        end
        subgraph "AV-WAN Baseline"
            direction TB
            AVWAN_RUN["av_wan/run.py"]:::green
            AVWAN_SENS["av_wan/avwan_sensors.py"]:::green
            AVWAN_MOD["av_wan/models"]:::green
            AVWAN_PPO["av_wan/ppo"]:::green
        end
        subgraph "SAVI Baseline"
            direction TB
            SAVI_RUN["savi/run.py"]:::green
            SAVI_MOD["savi/models"]:::green
            SAVI_PPO["savi/ppo & ddppo"]:::green
        end
    end

    CSIM -->|"observations"| AVNAV_RUN
    AVNAV_RUN -->|"feeds obs"| AVNAV_MOD
    AVNAV_MOD -->|"actions"| CSIM
    CSIM -->|"observations"| AVWAN_RUN
    AVWAN_RUN -->|"feeds obs"| AVWAN_MOD
    AVWAN_MOD -->|"actions"| CSIM
    CSIM -->|"observations"| SAVI_RUN
    SAVI_RUN -->|"feeds obs"| SAVI_MOD
    SAVI_MOD -->|"actions"| CSIM

    AVNAV_RUN --> BASE_TR
    AVWAN_RUN --> BASE_TR
    SAVI_RUN --> BASE_TR
    BASE_TR --> ROLL
    ROLL --> BASE_TR
    BASE_TR --> AVNAV_RUN
    BASE_TR --> AVWAN_RUN
    BASE_TR --> SAVI_RUN

    RUN --> AVNAV_RUN
    RUN --> AVWAN_RUN
    RUN --> SAVI_RUN

    %% Examples
    DEMO -.-> RUN
    DEMO -.-> CSIM
    subgraph "Examples / Tutorials"
        direction TB
        EX1["minimal_example.py"]:::orange
        EX2["soundspaces2_quick_tutorial.ipynb"]:::orange
    end
    RUN -.-> EX1
    RUN -.-> EX2

    %% External Dependencies
    subgraph "External Dependencies"
        HAB["Habitat API"]:::grey
        TORCH["PyTorch"]:::grey
        DOCKER["Docker (Dockerfile, .dockerignore)"]:::grey
    end
    SIM_PKG --> HAB
    BASE_TR --> TORCH
    RUN --> DOCKER

    %% Click Events
    click CSIM "https://github.com/raphael310/sound-spaces/blob/main/soundspaces/continuous_simulator.py"
    click SIM "https://github.com/raphael310/sound-spaces/blob/main/soundspaces/simulator.py"
    click DATA_LOAD "https://github.com/raphael310/sound-spaces/blob/main/soundspaces/datasets/audionav_dataset.py"
    click DATA_LOAD "https://github.com/raphael310/sound-spaces/blob/main/soundspaces/datasets/semantic_audionav_dataset.py"
    click TASKS "https://github.com/raphael310/sound-spaces/blob/main/soundspaces/tasks/audionav_task.py"
    click TASKS "https://github.com/raphael310/sound-spaces/blob/main/soundspaces/tasks/semantic_audionav_task.py"
    click TASKS "https://github.com/raphael310/sound-spaces/blob/main/soundspaces/tasks/action_space.py"
    click RES "https://github.com/raphael310/sound-spaces/blob/main/res/Material Map.txt"
    click RES "https://github.com/raphael310/sound-spaces/blob/main/res/singing.wav"
    click PANO "https://github.com/raphael310/sound-spaces/blob/main/PanoIR/render_panoIR.py"
    click DL "https://github.com/raphael310/sound-spaces/blob/main/scripts/download_data.py"
    click GEN_MP "https://github.com/raphael310/sound-spaces/blob/main/scripts/generate_matterport3d_metadata.py"
    click GEN_REP "https://github.com/raphael310/sound-spaces/blob/main/scripts/generate_replica_metadata.py"
    click AMB "https://github.com/raphael310/sound-spaces/blob/main/scripts/ambisonic_to_binaural.py"
    click CACHE "https://github.com/raphael310/sound-spaces/blob/main/scripts/cache_observations.py"
    click DEMO "https://github.com/raphael310/sound-spaces/blob/main/scripts/interactive_demo.py"
    click CFG "https://github.com/raphael310/sound-spaces/tree/main/configs/audionav"
    click CFG "https://github.com/raphael310/sound-spaces/tree/main/configs/semantic_audionav"
    click AVNAV_RUN "https://github.com/raphael310/sound-spaces/blob/main/ss_baselines/av_nav/run.py"
    click AVNAV_MOD "https://github.com/raphael310/sound-spaces/tree/main/ss_baselines/av_nav/models"
    click AVNAV_PPO "https://github.com/raphael310/sound-spaces/tree/main/ss_baselines/av_nav/ddppo"
    click AVNAV_PPO "https://github.com/raphael310/sound-spaces/tree/main/ss_baselines/av_nav/ppo"
    click AVWAN_RUN "https://github.com/raphael310/sound-spaces/blob/main/ss_baselines/av_wan/run.py"
    click AVWAN_SENS "https://github.com/raphael310/sound-spaces/blob/main/ss_baselines/av_wan/avwan_sensors.py"
    click AVWAN_MOD "https://github.com/raphael310/sound-spaces/tree/main/ss_baselines/av_wan/models"
    click AVWAN_PPO "https://github.com/raphael310/sound-spaces/tree/main/ss_baselines/av_wan/ppo"
    click SAVI_RUN "https://github.com/raphael310/sound-spaces/blob/main/ss_baselines/savi/run.py"
    click SAVI_MOD "https://github.com/raphael310/sound-spaces/tree/main/ss_baselines/savi/models"
    click SAVI_PPO "https://github.com/raphael310/sound-spaces/tree/main/ss_baselines/savi/ddppo"
    click SAVI_PPO "https://github.com/raphael310/sound-spaces/tree/main/ss_baselines/savi/ppo"
    click BASE_TR "https://github.com/raphael310/sound-spaces/blob/main/ss_baselines/common/base_trainer.py"
    click BASE_REG "https://github.com/raphael310/sound-spaces/blob/main/ss_baselines/common/baseline_registry.py"
    click ROLL "https://github.com/raphael310/sound-spaces/blob/main/ss_baselines/common/rollout_storage.py"
    click EX1 "https://github.com/raphael310/sound-spaces/blob/main/examples/minimal_example.py"
    click EX2 "https://github.com/raphael310/sound-spaces/blob/main/examples/soundspaces2_quick_tutorial.ipynb"
    click DOCKER "https://github.com/raphael310/sound-spaces/tree/main/Dockerfile"
    click DOCKER "https://github.com/raphael310/sound-spaces/blob/main/.dockerignore"

    %% Styles
    classDef blue fill:#D6EAF8,stroke:#3498DB,color:#000;
    classDef green fill:#D5F5E3,stroke:#27AE60,color:#000;
    classDef orange fill:#FCF3CF,stroke:#F39C12,color:#000;
    classDef grey fill:#F2F3F4,stroke:#7F8C8D,color:#000;
    class CFG,DEMOS,EX1,EX2 orange
    class SIM_PKG,CSIM,SIM,DATA_LOAD,TASKS blue
    class AVNAV_RUN,AVNAV_MOD,AVNAV_PPO,AVWAN_RUN,AVWAN_MOD,AVWAN_PPO,SAVI_RUN,SAVI_MOD,SAVI_PPO,BASE_TR,BASE_REG,ROLL green
    class RES,PANO,DATASETS,HAB,TORCH,DOCKER grey
```
