# storytold/realtime-voice-conversion architecture

> Architecture diagram of the GitHub repository storytold/realtime-voice-conversion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/storytold/realtime-voice-conversion
- Repository: https://github.com/storytold/realtime-voice-conversion

## Overview

This repository is a live voice-conversion application: Rust captures microphone audio, buffers and sends it over ZeroMQ using protobuf, while a Python sidecar converts speech with CycleGAN-VC and returns audio for playback. A separate browser demo exposes upload-based conversion. The graph also shows model preprocessing and network internals, plus the documented training/evaluation capability. Relationships reflect the excerpts and README; unsampled implementation details are omitted where their calls cannot be verified.

Main entry points: [`src/sidecar.rs`](https://github.com/storytold/realtime-voice-conversion/blob/master/src/sidecar.rs) (Rust sidecar client), [`src/audio_hardware.rs`](https://github.com/storytold/realtime-voice-conversion/blob/master/src/audio_hardware.rs) (Audio hardware), [`cycle_gan/sidecar.py`](https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/sidecar.py) (Python sidecar server), [`cycle_gan/demo_server.py`](https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/demo_server.py) (HTTP demo server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_capture["Audio capture and playback"]
  node_main["Audio runtime<br/>[main.rs]"]
  node_hardware["Audio hardware<br/>[audio_hardware.rs]"]
  node_queues["Audio queues<br/>[audio_queue.rs]"]
  node_synthesis["Audio synthesis<br/>[synthesis.rs]"]
end

subgraph group_transport["Sidecar transport"]
  node_rust_sidecar["Rust sidecar client<br/>[sidecar.rs]"]
  node_protocol["Audio messages<br/>[voder_audio.rs]"]
  node_python_sidecar["Python sidecar server<br/>[sidecar.py]"]
end

subgraph group_conversion["Speech conversion"]
  node_converter["Speech converter<br/>[sidecar.py]"]
  node_preprocess["Audio preprocessing<br/>[preprocess.py]"]
end

subgraph group_model["CycleGAN model"]
  node_model["CycleGAN inference and training<br/>[model.py]"]
  node_network["Generator and discriminator layers<br/>[module.py]"]
end

subgraph group_demo["Browser demo"]
  node_demo_server["HTTP demo server<br/>[demo_server.py]"]
end

node_speaker(("Microphone and speaker"))
node_demo_user(("Browser user"))

node_speaker -->|"provides audio"| node_hardware
node_main -->|"starts audio"| node_hardware
node_main -->|"starts client"| node_rust_sidecar
node_hardware -->|"buffers samples"| node_queues
node_rust_sidecar -->|"drains and fills"| node_queues
node_hardware -->|"plays audio"| node_speaker
node_rust_sidecar -->|"encodes messages"| node_protocol
node_rust_sidecar -->|"exchanges audio"| node_python_sidecar
node_python_sidecar -.->|"decodes messages"| node_protocol
node_python_sidecar -.->|"converts requests"| node_converter
node_converter -->|"processes speech"| node_preprocess
node_converter -->|"runs inference"| node_model
node_model -->|"builds networks"| node_network
node_demo_user -->|"uploads audio"| node_demo_server
node_demo_server -->|"loads and tests"| node_model
node_demo_server -->|"processes audio"| node_preprocess

click node_main "https://github.com/storytold/realtime-voice-conversion/blob/master/src/main.rs"
click node_hardware "https://github.com/storytold/realtime-voice-conversion/blob/master/src/audio_hardware.rs"
click node_queues "https://github.com/storytold/realtime-voice-conversion/blob/master/src/audio_queue.rs"
click node_synthesis "https://github.com/storytold/realtime-voice-conversion/blob/master/src/synthesis.rs"
click node_rust_sidecar "https://github.com/storytold/realtime-voice-conversion/blob/master/src/sidecar.rs"
click node_protocol "https://github.com/storytold/realtime-voice-conversion/blob/master/src/protos/voder_audio.rs"
click node_python_sidecar "https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/sidecar.py"
click node_converter "https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/sidecar.py"
click node_preprocess "https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/preprocess.py"
click node_model "https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/model.py"
click node_network "https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/module.py"
click node_demo_server "https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/demo_server.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_hardware,node_queues,node_synthesis,node_demo_user toneBlue
class node_rust_sidecar,node_protocol,node_python_sidecar toneAmber
class node_converter,node_preprocess toneMint
class node_model,node_network toneRose
class node_demo_server,node_speaker toneIndigo
```

## Components

### Audio capture and playback

- [Audio runtime](https://github.com/storytold/realtime-voice-conversion/blob/master/src/main.rs): `src/main.rs`
- [Audio hardware](https://github.com/storytold/realtime-voice-conversion/blob/master/src/audio_hardware.rs): `src/audio_hardware.rs`
- [Audio queues](https://github.com/storytold/realtime-voice-conversion/blob/master/src/audio_queue.rs): `src/audio_queue.rs`
- [Audio synthesis](https://github.com/storytold/realtime-voice-conversion/blob/master/src/synthesis.rs): `src/synthesis.rs`

### Sidecar transport

- [Rust sidecar client](https://github.com/storytold/realtime-voice-conversion/blob/master/src/sidecar.rs): `src/sidecar.rs`
- [Audio messages](https://github.com/storytold/realtime-voice-conversion/blob/master/src/protos/voder_audio.rs): `src/protos/voder_audio.rs`
- [Python sidecar server](https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/sidecar.py): `cycle_gan/sidecar.py`

### Speech conversion

- [Speech converter](https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/sidecar.py): `cycle_gan/sidecar.py`
- [Audio preprocessing](https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/preprocess.py): `cycle_gan/preprocess.py`

### CycleGAN model

- [CycleGAN inference and training](https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/model.py): `cycle_gan/model.py`
- [Generator and discriminator layers](https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/module.py): `cycle_gan/module.py`

### Browser demo

- [HTTP demo server](https://github.com/storytold/realtime-voice-conversion/blob/master/cycle_gan/demo_server.py): `cycle_gan/demo_server.py`

### Other components

- **Microphone and speaker**
- **Browser user**

## Connections

- Microphone and speaker → Audio hardware: provides audio (evidence: `src/audio_hardware.rs`)
- Audio runtime → Audio hardware: starts audio (evidence: `src/main.rs`)
- Audio runtime → Rust sidecar client: starts client (evidence: `src/main.rs`)
- Audio hardware → Audio queues: buffers samples (evidence: `src/audio_hardware.rs`)
- Rust sidecar client → Audio queues: drains and fills (evidence: `src/sidecar.rs`)
- Audio hardware → Microphone and speaker: plays audio (evidence: `src/audio_hardware.rs`)
- Rust sidecar client → Audio messages: encodes messages (evidence: `src/sidecar.rs`)
- Rust sidecar client → Python sidecar server: exchanges audio (evidence: `README.md`)
- Python sidecar server → Audio messages: decodes messages
- Python sidecar server → Speech converter: converts requests (evidence: `cycle_gan/sidecar.py`)
- Speech converter → Audio preprocessing: processes speech (evidence: `cycle_gan/sidecar.py`)
- Speech converter → CycleGAN inference and training: runs inference (evidence: `cycle_gan/sidecar.py`)
- CycleGAN inference and training → Generator and discriminator layers: builds networks (evidence: `cycle_gan/model.py`)
- Browser user → HTTP demo server: uploads audio (evidence: `cycle_gan/demo_server.py`)
- HTTP demo server → CycleGAN inference and training: loads and tests (evidence: `cycle_gan/demo_server.py`)
- HTTP demo server → Audio preprocessing: processes audio (evidence: `cycle_gan/demo_server.py`)
