How peiwensun2000/both-ears-wide-open works
Project Type & Purpose
• A research‐oriented, multi‐modal spatial audio generation toolkit.
• Three main pillars:
– BEWO‐1M Dataset (data preparation & storage)
– Inference (text‐to‐audio generation via pretrained models)
– Evaluation (metric calculations, visualization)
Top‐Level Components
A. Data Subsystem
• “datasets/” (scripts & metadata for BEWO‐1M)
• “evaluations/Dataset/…” (download, process, CSV creation)
• Ground‐truth media storage (audio files, frames, meta.json)
B. Model Subsystem
• “models/bewo_config/” (checkpoint + config JSON)
• “models/simple_generation.py” (direct text→audio)
• “models/gpt_induction.py” (coarse‐to‐fine via GPT/DeepSeek)
• “models/stable_audio_tools/” (core library for encoders, diffusion, transformers)
C. Evaluation Subsystem
• “evaluations/main.py” & “automate_process.py” (entrypoints)
• “evaluations/data/…” (data loaders for each dataset)
• “evaluations/models/…” (audio & AV encoders, loss modules)
• “evaluations/cal_avg_*.py” (metric aggregators: ITD, FAD, mean)
• “evaluations/scripts/” (bash scripts for training, eval, visualization)
D. External Services & Infra
• Huggingface Datasets & Model Hub
• GPT‐4o / DeepSeek API for attribute induction
• Conda/Python environment (environment.yml)
• CUDA/GPU compute
Relationships & Data Flows
1. Data Preparation
• Raw media → processing scripts → standardized CSV + directories
• Data stored locally (datasets/) or on Huggingface
2. Model Inference
• User CLI (simple_generation.py or gpt_induction.py)
• Loads checkpoint + model_config
• (Optional) Calls GPT/DeepSeek service → returns spatial attributes
• Passes prompt + attributes to stable_audio_tools pipeline
• Outputs binaural audio file
3. Evaluation
• Load test set via data loaders
• Run pretrained models (audio encoders)
• Compute metrics (ITD, FAD, etc.)
• Produce CSV + visualizations via vis_scripts
Architectural Patterns & Principles
• Modular separation:
• Data layer (all readers/preprocessors)
• Model layer (inference & core ML library)
• Evaluation layer (metrics, scripts, visualization)
• CLI‐driven workflows (entrypoints invoke specific pipelines)
• Reuse of stable_audio_tools: library + training/inference split
• External API abstraction for induction (GPT vs. manual flags)
Final Checklist
– Clear component labels
– External dependencies called out separately