How storytold/realtime-voice-conversion works
This repository is a live voice-conversion application: Rust captures microphone audio, buffers and sends it over ZeroMQ using protobuf, while a Python sidecar converts speech with CycleGAN-VC and returns audio for playback. A separate browser demo exposes upload-based conversion. The graph also shows model preprocessing and network internals, plus the documented training/evaluation capability. Relationships reflect the excerpts and README; unsampled implementation details are omitted where their calls cannot be verified.
Main entry points: sidecar.rs Rust sidecar client · audio_hardware.rs Audio hardware · sidecar.py Python sidecar server · demo_server.py HTTP demo server
Components 14
Audio capture and playback
- Audio runtime
src/main.rs - Audio hardware
src/audio_hardware.rs - Audio queues
src/audio_queue.rs - Audio synthesis
src/synthesis.rs
Sidecar transport
- Rust sidecar client
src/sidecar.rs - Audio messages
src/protos/voder_audio.rs - Python sidecar server
cycle_gan/sidecar.py
Speech conversion
- Speech converter
cycle_gan/sidecar.py - Audio preprocessing
cycle_gan/preprocess.py
CycleGAN model
- CycleGAN inference and training
cycle_gan/model.py - Generator and discriminator layers
cycle_gan/module.py
Browser demo
- HTTP demo server
cycle_gan/demo_server.py
Other components
- Microphone and speaker
- Browser user
Connections 16
- Microphone and speaker → Audio hardware: provides audio
src/audio_hardware.rs - Audio runtime → Audio hardware: starts audio
src/main.rs - Audio runtime → Rust sidecar client: starts client
src/main.rs - Audio hardware → Audio queues: buffers samples
src/audio_hardware.rs - Rust sidecar client → Audio queues: drains and fills
src/sidecar.rs - Audio hardware → Microphone and speaker: plays audio
src/audio_hardware.rs - Rust sidecar client → Audio messages: encodes messages
src/sidecar.rs - Rust sidecar client → Python sidecar server: exchanges audio
README.md - Python sidecar server → Audio messages: decodes messages
- Python sidecar server → Speech converter: converts requests
cycle_gan/sidecar.py - Speech converter → Audio preprocessing: processes speech
cycle_gan/sidecar.py - Speech converter → CycleGAN inference and training: runs inference
cycle_gan/sidecar.py - CycleGAN inference and training → Generator and discriminator layers: builds networks
cycle_gan/model.py - Browser user → HTTP demo server: uploads audio
cycle_gan/demo_server.py - HTTP demo server → CycleGAN inference and training: loads and tests
cycle_gan/demo_server.py - HTTP demo server → Audio preprocessing: processes audio
cycle_gan/demo_server.py