How w-okada/voice-changer works
Project type and purpose
• This is a full-stack, multi-platform real-time voice conversion application (VCClient) with both local (Web Audio API/worklet) and network-offloaded (client↔server) modes. It includes:
• A web/desktop client (React+TypeScript, Webpack) under client/, a shared TypeScript library for audio worklet under client/lib/, and a minimal Python client under client/python/.
• A Python server (Flask/Socket.IO) under server/ exposing REST endpoints (server/restapi/) and WebSocket audio‐streaming (server/sio/) that dispatches audio data to a plugin-style “Voice Conversion Engine” supporting multiple model types (RVC, DiffusionSVC, EasyVC, etc.).
• A standalone recorder UI (React+TS) under recorder/.
• Trainer components and scripts under trainer/ & docker_trainer/ for model training.
• Docker setups (docker/, docker_vcclient/, docker_trainer/) and helper scripts under script/.
• Documentation (docs/, tutorials/) and CI workflows (.github/).
File structure
• Top-level architectural slices:
• client/ (UI and audio pipeline in browser/desktop)
• server/ (backend services: REST API, Socket.IO, voice-changer modules)
• recorder/ (corpus recording front end)
• trainer/ + docker_trainer/ (model training pipelines)
• docker*/, script/ (containerization & deployment)
• docs/, tutorials/ (user/developer docs)
• Key patterns:
• Component-based React frontends (client/demo/src/components, recorder/src/100_components)
• Shared TypeScript library for audio handling & worklet nodes (client/lib/)
• Plugin-style server modules in server/voice_changer/, dynamically loaded by VoiceChangerManager
• Dual communication paths: HTTP/REST for metadata and file upload/download; WebSocket (Socket.IO) for real-time audio streams
• Docker images per role (vcclient, trainer, vcclient network mode)
README for additional insights
• The README emphasizes:
• Multi-AI-model support (RVC, Beatrice v1/v2, MMVC, etc.) via plugin modules.
• Standalone (local) and network modes (offload CPU/GPU to server)
• Cross-platform support (Windows/Mac/Linux/Colab)
• A REST API for programmatic control and a Socket.IO layer for streaming.
a. Main components
1. Client UI (React/TS browser or Electron desktop)
2. Client audio pipeline (Web Audio API → VoiceChangerWorkletNode)
3. Client networking layer (REST client, Socket.IO client)
4. Server API layer (Flask-based REST endpoints + MMVC_Rest etc.)
5. Server real-time stream layer (Socket.IO namespace under server/sio/)
6. Voice Conversion Engine
• Plugin registry loading modules under server/voice_changer/
• Individual modules for each AI model (RVC, DiffusionSVC, etc.)
7. Model repository/storage (server/data/, server/model_dir_static/)
8. Trainer pipeline (PyTorch/ONNX conversion under trainer/, docker_trainer/)
9. Recorder UI (separate React app under recorder/)
10. Docker containers (vcclient, vcclient network mode, trainer)
11. Documentation portal (docs/)
b. Relationships & interactions
• UI triggers mic capture → feeds into Web Audio worklet → (local mode) worklet processes audio via client/lib → output to speakers
• (Network mode) worklet sends PCM chunks over Socket.IO to the server stream namespace
• Server receives audio chunks → VoiceChangerManager dispatches to the correct model plugin → processed chunks streamed back over Socket.IO → client plays audio
• REST endpoints used for: model listing/download (MMVC_Rest_Fileuploader), slot management, server info, file uploads, error handling
• Trainer pipelines triggered separately in dedicated containers, produce new model artifacts vs. onnx exporters
c. Architectural patterns/design principles
• Micro-frontend plus micro-library: separate demo, recorder, and Python clients, sharing core audio-pipeline TS library
• Plugin architecture on server for voice conversion algorithms
• Dual-mode operation: local worklet vs. network offload
• Containerization per role, CI/CD workflows in GitHub Actions
d. Key technologies/frameworks
• Frontend: React, TypeScript, Webpack, PostCSS
• Audio: Web Audio API, AudioWorkletNode
• Networking: Axios/Fetch for REST, Socket.IO for streaming
• Backend: Python, Flask (or equivalent), python-socketio
• ML frameworks: PyTorch, ONNX, custom graph inference
• Docker, shell scripts for deployment/training
• Use clear labels for each component (e.g., “VCClient UI”, “VoiceChangerWorkletNode”, “Socket.IO Stream” etc.)
• Audio data flow (mic → worklet → network → server engine → network → worklet → speakers)
• Control/metadata flow (UI → REST → server; server → REST → UI)
• UI components (blue)
• Audio pipeline (green)
• Networking (orange)
• Server backend (red)
• Storage & training (gray)
• Frontend apps (rounded rectangles)
• Backend services/APIs (rectangles)