How babysor/mockingbird works
MockingBird is a voice-cloning and voice-conversion system with command-line, desktop-toolbox, and web interfaces. In the principal workflow, a user supplies speech and a target voice; the encoder derives a speaker representation, the synthesizer generates mel spectrograms from text and that representation, and a vocoder produces waveform audio. Voice conversion adds PPG extraction and PPG-to-mel conversion. Separate documented workflows preprocess datasets and train encoder, synthesizer, PPG-to-mel, and vocoder models. The graph includes these documented capabilities; several training and web relationships are inferred from README descriptions because their implementation was not sampled.
Main entry points: ui.py Desktop toolbox · app_vc.py Conversion app · preprocess.py Dataset preprocessing · app.py Web synthesis app
Components 19
User interfaces
- Desktop toolbox
control/toolbox/ui.py - Web interface
control/mkgui/base/ui/streamlit_ui.py - Web synthesis app
control/mkgui/app.py - Conversion app
control/mkgui/app_vc.py
Voice generation
- Speaker encoder
models/encoder/inference.py - Text synthesizer
models/synthesizer/inference.py - HiFi-GAN vocoder
models/vocoder/hifigan/inference.py - Fre-GAN vocoder
models/vocoder/fregan/inference.py - WaveRNN vocoder
models/vocoder/wavernn/inference.py
Voice conversion
- PPG extractor
models/ppg_extractor - PPG-to-mel decoder
models/ppg2mel/rnn_decoder_mol.py
Data and training
- Dataset preprocessing
models/synthesizer/preprocess.py - Encoder training
models/encoder/train.py - Synthesizer training
models/synthesizer/train.py - PPG-mel training
models/ppg2mel/train/loss.py - Vocoder training
models/vocoder/hifigan/train.py
Shared audio
- Audio utilities
models/encoder/audio.py - Generated audio
Other components
- User
Connections 27
- User → Desktop toolbox: submits speech
- User → Web interface: submits request
- Web interface → Web synthesis app: routes request
control/mkgui/base/ui/streamlit_ui.py - Web synthesis app → Speaker encoder: loads model
control/mkgui/app.py - Web synthesis app → Text synthesizer: loads model
control/mkgui/app.py - Web synthesis app → HiFi-GAN vocoder: loads model
control/mkgui/app.py - Desktop toolbox → Speaker encoder: uses encoder
control/toolbox/__init__.py - Desktop toolbox → Text synthesizer: uses synthesizer
control/toolbox/__init__.py - Desktop toolbox → HiFi-GAN vocoder: uses vocoder
control/toolbox/__init__.py - Desktop toolbox → Fre-GAN vocoder: uses vocoder
control/toolbox/__init__.py - Desktop toolbox → WaveRNN vocoder: uses vocoder
control/toolbox/__init__.py - Speaker encoder → Audio utilities: preprocesses speech
models/encoder/inference.py - Text synthesizer → Generated audio: generates spectrogram
- HiFi-GAN vocoder → Generated audio: generates waveform
models/vocoder/hifigan/inference.py - Fre-GAN vocoder → Generated audio: generates waveform
models/vocoder/fregan/inference.py - WaveRNN vocoder → Generated audio: generates waveform
models/vocoder/wavernn/inference.py - User → Conversion app: requests conversion
- Conversion app → PPG extractor: extracts PPG
control/mkgui/app_vc.py - Conversion app → PPG-to-mel decoder: converts features
- Conversion app → Speaker encoder: loads encoder
control/mkgui/app_vc.py - Conversion app → HiFi-GAN vocoder: loads vocoder
control/mkgui/app_vc.py - PPG-to-mel decoder → Generated audio: produces mel
- Dataset preprocessing → Synthesizer training: prepares data
- Dataset preprocessing → Encoder training: prepares data
- Dataset preprocessing → PPG-mel training: prepares data
- Dataset preprocessing → Vocoder training: prepares data
- User → Dataset preprocessing: provides dataset