How dinhthixuanbinh/voicefilter works
Project type and purpose
• This is a machine-learning research project (TensorFlow/Keras implementation) for speech separation (“VoiceFilter”) targeted at isolating a primary speaker’s voice from a mixed audio stream.
• It ingests LibriSpeech data, builds a custom dataset (spectrograms + speaker embeddings), trains a deep neural network, and evaluates restoration quality (SDR).
File-structure analysis
• Top level:
• README.md, VoiceFilter.ipynb – documentation and exploratory notebook
• requirements.txt – Python dependencies
• configuration/ – Python modules defining audio I/O parameters (Audio.py) and hyperparameters (HyperParams.py)
• dataset/ – scripts to generate and load the dataset:
• dataset_creation.py (mixing, reference embedding)
• directory_structure.py (folder organization)
• preload_training_dataset.py (data generators)
• speech_collection.py (Librispeech subset selection)
• model/ – core network definition:
• model.py (functional Keras model)
• sequence_generator.py (time-series input handling)
• train_test/ – orchestration scripts:
• train.py (training loop, callbacks)
• testing.py (evaluation, SDR measurement)
Key insights from README
• Two-level architecture:
a. Data-prep pipeline: converting audio → magnitude spectrogram + d-vector embedding.
b. Model pipeline: takes spectrogram + d-vector → mask → output spectrogram → reconstruct audio.
• External dependency: pre-trained speaker embedding extractor (from mindslab-ai).
• Metrics: MSE loss, Adam optimizer, evaluation via SDR.
• Deployment/environment: Python 3.6.9, Google Colab / GPU.
a. Main components
• Data sources:
– LibriSpeech raw audio
– Pre-trained embedding model (external service or library)
• Data preprocessing pipeline:
– Audio Loader & Mixer
– Spectrogram converter (librosa)
– Embedding generator
– Custom Keras data generator
• Model training pipeline:
– Keras functional model (model.py)
– Hyperparameters module (HyperParams.py)
– Training scripts with callbacks
– Checkpoint & logging
• Evaluation pipeline:
– Testing script computing SDR
– Audio reconstructor (spectrogram + phase)
• Artifacts & assets:
– Stored model weights
– Audio samples (assets/audio_samples)
b. Relationships and data flows
• From LibriSpeech → dataset_creation.py → spectrograms
• From spectrograms + embedding extractor → data generator → model training
• From model → mask → audio reconstructor → output audio
• From trained model → testing.py → SDR metric
• Feedback loop: training metrics → hyperparameter tuning.
c. Architectural patterns or principles
• Modular separation of concerns:
• configuration vs. data vs. model vs. orchestration
• Pipeline pattern for data preprocessing → training → evaluation
• Use of custom generators and callbacks (demonstrating extensibility).
d. Key technologies
• Python 3.6, TensorFlow/Keras
• Librosa (audio processing)
• NumPy, multiprocessing for data loading
• Jupyter Notebook for exploration