# peiwensun2000/both-ears-wide-open architecture

> Architecture diagram of the GitHub repository peiwensun2000/both-ears-wide-open, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/peiwensun2000/both-ears-wide-open
- Repository: https://github.com/peiwensun2000/both-ears-wide-open

## Overview

##### Project Type & Purpose
- A research‐oriented, multi‐modal spatial audio generation toolkit.
- Three main pillars:
– BEWO‐1M Dataset (data preparation & storage)
– Inference (text‐to‐audio generation via pretrained models)
– Evaluation (metric calculations, visualization)

##### Top‐Level Components
A. Data Subsystem
- “datasets/” (scripts & metadata for BEWO‐1M)
- “evaluations/Dataset/…” (download, process, CSV creation)
- Ground‐truth media storage (audio files, frames, meta.json)

B. Model Subsystem
- “models/bewo_config/” (checkpoint + config JSON)
- “models/simple_generation.py” (direct text→audio)
- “models/gpt_induction.py” (coarse‐to‐fine via GPT/DeepSeek)
- “models/stable_audio_tools/” (core library for encoders, diffusion, transformers)

C. Evaluation Subsystem
- “evaluations/main.py” & “automate_process.py” (entrypoints)
- “evaluations/data/…” (data loaders for each dataset)
- “evaluations/models/…” (audio & AV encoders, loss modules)
- “evaluations/cal_avg_*.py” (metric aggregators: ITD, FAD, mean)
- “evaluations/scripts/” (bash scripts for training, eval, visualization)

D. External Services & Infra
- Huggingface Datasets & Model Hub
- GPT‐4o / DeepSeek API for attribute induction
- Conda/Python environment (environment.yml)
- CUDA/GPU compute

##### Relationships & Data Flows
1. Data Preparation
- Raw media → processing scripts → standardized CSV + directories
- Data stored locally (datasets/) or on Huggingface

2. Model Inference
- User CLI (simple_generation.py or gpt_induction.py)
- Loads checkpoint + model_config
- (Optional) Calls GPT/DeepSeek service → returns spatial attributes
- Passes prompt + attributes to stable_audio_tools pipeline
- Outputs binaural audio file

3. Evaluation
- Load test set via data loaders
- Run pretrained models (audio encoders)
- Compute metrics (ITD, FAD, etc.)
- Produce CSV + visualizations via vis_scripts

##### Architectural Patterns & Principles
- Modular separation:
- **Data layer** (all readers/preprocessors)
- **Model layer** (inference & core ML library)
- **Evaluation layer** (metrics, scripts, visualization)
- CLI‐driven workflows (entrypoints invoke specific pipelines)
- Reuse of stable_audio_tools: library + training/inference split
- External API abstraction for induction (GPT vs. manual flags)

##### Final Checklist
– Clear component labels
– External dependencies called out separately

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Subsystem
    subgraph "Data Subsystem"
        D1["BEWO-1M Dataset\n(scripts & metadata)"]:::data
        D2["Dataset Processing\n(ground-truth media ingest & CSV creation)"]:::data
    end

    %% Model Subsystem
    subgraph "Model Subsystem"
        M1["Simple Text→Audio CLI"]:::model
        M2["GPT-based Coarse-to-Fine CLI"]:::model
        M3["Pretrained Checkpoints & Configs"]:::model
        subgraph "Core Audio Generation Library"
            ML1["AudioEncoder"] 
            ML2["AVEncoder"] 
            ML3["Diffusion"] 
            ML4["Transformer"] 
            ML5["Codebook"] 
        end
    end

    %% Evaluation Subsystem
    subgraph "Evaluation Subsystem"
        subgraph "Entrypoints"
            E1A["main.py"]:::eval
            E1B["automate_process.py"]:::eval
        end
        E2["Evaluation Dataset Loaders"]:::eval
        E3["Evaluation Models & Loss Modules"]:::eval
        subgraph "Metric Aggregators"
            AG1["cal_avg.py"]:::eval
            AG2["cal_avg_idt_fad.py"]:::eval
            AG3["cal_avg_idt_fad_uneq.py"]:::eval
            AG4["cal_avg_mean_itd.py"]:::eval
            AG5["cal_avg_mean_itd_uneq.py"]:::eval
            AG6["cal_avg_mean_only.py"]:::eval
        end
        E4["Evaluation Scripts"]:::eval
        E5["Visualization Scripts"]:::eval
    end

    %% External Services & Infrastructure
    subgraph "External Services & Infrastructure"
        EX1["Huggingface Datasets & Model Hub"]:::ext
        EX2["GPT-4o / DeepSeek API"]:::ext
        EX3["Conda/Python Environment Config"]:::ext
        EX4["CUDA/GPU Compute"]:::ext
    end

    %% Connections - Data to Model
    D1 -->|reads raw media| D2
    D2 -->|CSV + media paths| M1
    D2 -->|CSV + media paths| M2
    D2 -->|upload/download| EX1

    %% Connections - Model Inference
    M1 -->|"loads checkpoint+config"| M3
    M2 -->|"loads checkpoint+config"| M3
    EX1 -->|"download checkpoint"| M3
    M2 -->|"calls API"| EX2
    EX2 -->|"attributes"| M2
    M1 -->|"text prompt"| ML1
    M2 -->|"prompt+attributes"| ML1
    ML1 --> ML2 --> ML3 --> ML4 --> ML5
    ML5 -->|"binaural audio"| E1A
    ML5 -->|"binaural audio"| E1B

    %% Connections - Evaluation
    E1A -->|"load test set"| E2
    E1B -->|"load test set"| E2
    E1A -->|"run pretrained models"| E3
    E1B -->|"run pretrained models"| E3
    E1A -->|"compute metrics"| AG1
    E1A -->|"compute metrics"| AG2
    E1A -->|"compute metrics"| AG3
    E1A -->|"compute metrics"| AG4
    E1A -->|"compute metrics"| AG5
    E1A -->|"compute metrics"| AG6
    E1B -->|"compute metrics"| AG1
    E1B -->|"compute metrics"| AG2
    E1B -->|"compute metrics"| AG3
    E1B -->|"compute metrics"| AG4
    E1B -->|"compute metrics"| AG5
    E1B -->|"compute metrics"| AG6
    AG1 --> E5
    AG2 --> E5
    AG3 --> E5
    AG4 --> E5
    AG5 --> E5
    AG6 --> E5
    E1A --> E4
    E1B --> E4
    E4 --> E5

    %% Click Events
    click D1 "https://github.com/peiwensun2000/both-ears-wide-open/tree/main/datasets/"
    click D2 "https://github.com/peiwensun2000/both-ears-wide-open/tree/main/evaluations/Dataset/"
    click M1 "https://github.com/peiwensun2000/both-ears-wide-open/blob/main/models/simple_generation.py"
    click M2 "https://github.com/peiwensun2000/both-ears-wide-open/blob/main/models/gpt_induction.py"
    click M3 "https://github.com/peiwensun2000/both-ears-wide-open/tree/main/models/bewo_config/"
    click EX3 "https://github.com/peiwensun2000/both-ears-wide-open/blob/main/evaluations/environment.yml"
    click E1A "https://github.com/peiwensun2000/both-ears-wide-open/blob/main/evaluations/main.py"
    click E1B "https://github.com/peiwensun2000/both-ears-wide-open/blob/main/evaluations/automate_process.py"
    click E2 "https://github.com/peiwensun2000/both-ears-wide-open/tree/main/evaluations/data/"
    click E3 "https://github.com/peiwensun2000/both-ears-wide-open/tree/main/evaluations/models/"
    click AG1 "https://github.com/peiwensun2000/both-ears-wide-open/blob/main/evaluations/cal_avg.py"
    click AG2 "https://github.com/peiwensun2000/both-ears-wide-open/blob/main/evaluations/cal_avg_idt_fad.py"
    click AG3 "https://github.com/peiwensun2000/both-ears-wide-open/blob/main/evaluations/cal_avg_idt_fad_uneq.py"
    click AG4 "https://github.com/peiwensun2000/both-ears-wide-open/blob/main/evaluations/cal_avg_mean_itd.py"
    click AG5 "https://github.com/peiwensun2000/both-ears-wide-open/blob/main/evaluations/cal_avg_mean_itd_uneq.py"
    click AG6 "https://github.com/peiwensun2000/both-ears-wide-open/blob/main/evaluations/cal_avg_mean_only.py"
    click E4 "https://github.com/peiwensun2000/both-ears-wide-open/tree/main/evaluations/scripts/"
    click E5 "https://github.com/peiwensun2000/both-ears-wide-open/tree/main/evaluations/vis_scripts/"

    %% Styles
    classDef data fill:#d4f8d4,stroke:#333
    classDef model fill:#d4e1f9,stroke:#333
    classDef eval fill:#ffe0b3,stroke:#333
    classDef ext fill:#e0e0e0,stroke:#333
```
