How modelscope/clearervoice-studio works
Project Type and Purpose
• ClearerVoice‐Studio is not a single web app but a modular AI speech‐processing toolkit.
• It comprises three top‐level components:
1. ClearVoice – A unified inference library/CLI (supports denoising, separation, super‐resolution, speaker extraction).
2. Train – A collection of training & data-gen pipelines for multiple tasks.
3. SpeechScore – A speech quality‐assessment toolkit.
• README confirms pre-trained models, demos (HuggingFace/ModelScope/Streamlit), training scripts, and evaluation metrics.
File Structure
• Top‐level folders:
• clearvoice/ – library code, configs, dataloaders, models, network wrapper, demos, samples, Streamlit app.
• train/ – subfolders per task: speech_enhancement, speech_separation, speech_super_resolution, target_speaker_extraction, target_speaker_extraction_online, plus data_generation scripts.
• speechscore/ – metric modules, demo.
• Config files live under each component’s config/inference and config/train.
• Scripts: CLI demos (demo.py, demo_Numpy2Numpy.py, streamlit_app.py), training & inference shells (*.sh), train.py, solver.py.
• Data: sample audio/video in clearvoice/samples, training data SCPS, checkpoints under train subfolders.
README for Additional Insights
• Three main pillars: ClearVoice (inference), Train (finetune/train), SpeechScore (evaluation).
• Stack: Python, PyTorch, FFmpeg (for audio codecs), ONNX for some models, HuggingFace/ModelScope integration.
• Tasks supported: enhancement, separation, super‐resolution, speaker extraction (audio, audio-visual, gesture, EEG).
a. Main Components
• User Interfaces / Clients: CLI scripts, Streamlit app, HuggingFace & ModelScope web demos (external).
• ClearVoice Inference Library:
– Config loader (YAML/JSON)
– Data I/O & Dataloader
– Network wrapper & model factory
– Model implementations (FRCRN, MossFormer*, GAN, etc.)
– Utils (decode, video_process)
• Training Pipelines (train/…):
– Data generation scripts (noisy/reverb)
– Task‐specific dataloaders & datasets
– Model definitions & wrappers
– Networks module & Solver
– Training & inference scripts
• SpeechScore Module:
– CLI interface
– Metric implementations (PESQ, STOI, DNSMOS, SI-SDR, etc.)
• Data Stores: sample data, training datasets, pre-trained checkpoints.
• External Dependencies & Services: FFmpeg, PyTorch, HuggingFace Hub, ModelScope, ONNX runtime.
b. Relationships & Data Flows
1. Inference Flow:
User CLI/Streamlit → ClearVoice.Config → Dataloader (audio/video) → NetworkWrapper → Model → Output audio/Numpy → (optional) SpeechScore metrics
2. Training Flow:
DataGen scripts → Generated datasets → Task‐specific dataloader → Model definitions → Solver → Checkpoint
3. Evaluation Flow:
Trained model output → SpeechScore metrics → Result logs/reports
c. Architectural Patterns & Principles
– Layered Architecture:
Presentation (CLI/Streamlit), Application (ClearVoice API / Train scripts), Domain (model implementations), Infrastructure (dataloaders, utils, externals).
– Factory/Registry: network_wrapper dynamically instantiates models based on config.
– Modularity & Single‐Responsibility: each task & component lives in its own folder with isolated configs.
– Pluggability: new models or tasks can be added under clearvoice/models or train/… with minimal coupling.
d. Key Technologies
– Python 3.8+
– PyTorch & ONNX
– FFmpeg for codec support
– YAML/JSON config parsing
– CLI & Streamlit for front‐end demos
– HuggingFace & ModelScope integration