How gabrielmittag/nisqa works
Main Components
• Environment & Configuration:
• The project uses an Anaconda environment defined in “env.yml” and several YAML configuration files in the “config” directory (e.g., finetune, train_nisqa_cnn_sa_ap.yaml, train_nisqa_double_ended.yaml) that specify training and finetuning parameters.
• Core Library:
• The “nisqa” directory includes key modules like “NISQA_model.py” (which implements the deep learning model architectures) and “NISQA_lib.py” (providing helper functions).
• Executable Scripts:
• There are three main scripts:
• run_predict.py – for inference on single files, directories, or CSV-based inputs.
• run_train.py – for training new models or using transfer learning/finetuning.
• run_evaluate.py – for model evaluation, replicating tests from the NISQA paper.
• Pre-trained Model Weights:
• The “weights” directory contains the pre-trained weights (nisqa.tar for full quality, nisqa_mos_only.tar for single output, and nisqa_tts.tar for TTS naturalness).
Architectural Relationships and Data Flow
• Data Input & Preprocessing:
• Users provide speech data (WAV files or CSV listings) that are fed into the prediction or training pipelines.
• The configuration files define the mapping between data directories, CSV columns, and target labels.
• Model Execution:
• The prediction process (via run_predict.py) loads model weights from the “weights” directory and utilizes functions from the “nisqa” library to process and predict speech quality.
• For training, run_train.py uses the configurations from YAML files to orchestrate model initialization (with architectures that can include CNN, LSTM, Self-Attention, and various pooling methods) and to drive the training loop.
• Output & Reporting:
• Prediction results are output to the console and can also be saved as CSV files.
• During training and evaluation, logs and model checkpoints (stored in the “output_dir” specified in the configuration) are generated.
Architectural Patterns and Design Principles
• Modularity:
• The system is clearly partitioned into data ingestion, model computation (segmented logically as framewise processing, time-dependency, and pooling), and results reporting.
• Configurability:
• The YAML configuration files illustrate that the project is designed to easily swap different deep learning components (e.g., different combinations of CNN, LSTM, Self-Attention) and adapt for diverse training tasks (single-ended vs. double-ended models).
• Command-Line Driven Workflow:
• Different scripts are dedicated to prediction, training, and evaluation, which simplifies maintenance and testing.
Relevant Technologies, Frameworks, and Libraries
• Deep Learning Framework:
• Scripting and Configuration:
• YAML is used for experiment configuration.
• Data Formats:
• The project works with audio files (WAV format) and CSV for dataset management.