# modelscope/clearervoice-studio architecture

> Architecture diagram of the GitHub repository modelscope/clearervoice-studio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/modelscope/clearervoice-studio
- Repository: https://github.com/modelscope/clearervoice-studio

## Overview

##### Project Type and Purpose
- ClearerVoice‐Studio is **not** a single web app but a **modular AI speech‐processing toolkit**.
- It comprises three top‐level components:
1. **ClearVoice** – A unified inference library/CLI (supports denoising, separation, super‐resolution, speaker extraction).
2. **Train** – A collection of training & data-gen pipelines for multiple tasks.
3. **SpeechScore** – A speech quality‐assessment toolkit.
- README confirms pre-trained models, demos (HuggingFace/ModelScope/Streamlit), training scripts, and evaluation metrics.

##### File Structure
- Top‐level folders:
- **clearvoice/** – library code, configs, dataloaders, models, network wrapper, demos, samples, Streamlit app.
- **train/** – subfolders per task: speech_enhancement, speech_separation, speech_super_resolution, target_speaker_extraction, target_speaker_extraction_online, plus data_generation scripts.
- **speechscore/** – metric modules, demo.
- Config files live under each component’s `config/inference` and `config/train`.
- Scripts: CLI demos (`demo.py`, `demo_Numpy2Numpy.py`, `streamlit_app.py`), training & inference shells (`*.sh`), `train.py`, `solver.py`.
- Data: sample audio/video in `clearvoice/samples`, training data SCPS, checkpoints under train subfolders.

##### README for Additional Insights
- Three main pillars: ClearVoice (inference), Train (finetune/train), SpeechScore (evaluation).
- Stack: Python, PyTorch, FFmpeg (for audio codecs), ONNX for some models, HuggingFace/ModelScope integration.
- Tasks supported: enhancement, separation, super‐resolution, speaker extraction (audio, audio-visual, gesture, EEG).

a. **Main Components**
- User Interfaces / Clients: CLI scripts, Streamlit app, HuggingFace & ModelScope web demos (external).
- ClearVoice Inference Library:
– Config loader (YAML/JSON)
– Data I/O & Dataloader
– Network wrapper & model factory
– Model implementations (FRCRN, MossFormer*, GAN, etc.)
– Utils (decode, video_process)
- Training Pipelines (train/…):
– Data generation scripts (noisy/reverb)
– Task‐specific dataloaders & datasets
– Model definitions & wrappers
– Networks module & Solver
– Training & inference scripts
- SpeechScore Module:
– CLI interface
– Metric implementations (PESQ, STOI, DNSMOS, SI-SDR, etc.)
- Data Stores: sample data, training datasets, pre-trained checkpoints.
- External Dependencies & Services: FFmpeg, PyTorch, HuggingFace Hub, ModelScope, ONNX runtime.

b. **Relationships & Data Flows**
1. **Inference Flow**:
User CLI/Streamlit → ClearVoice.Config → Dataloader (audio/video) → NetworkWrapper → Model → Output audio/Numpy → (optional) SpeechScore metrics
2. **Training Flow**:
DataGen scripts → Generated datasets → Task‐specific dataloader → Model definitions → Solver → Checkpoint
3. **Evaluation Flow**:
Trained model output → SpeechScore metrics → Result logs/reports

c. **Architectural Patterns & Principles**
– **Layered Architecture**:
Presentation (CLI/Streamlit), Application (ClearVoice API / Train scripts), Domain (model implementations), Infrastructure (dataloaders, utils, externals).
– **Factory/Registry**: network_wrapper dynamically instantiates models based on config.
– **Modularity & Single‐Responsibility**: each task & component lives in its own folder with isolated configs.
– **Pluggability**: new models or tasks can be added under clearvoice/models or train/… with minimal coupling.

d. **Key Technologies**
– Python 3.8+
– PyTorch & ONNX
– FFmpeg for codec support
– YAML/JSON config parsing
– CLI & Streamlit for front‐end demos
– HuggingFace & ModelScope integration

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Services
    HF["HuggingFace/ModelScope Web Demos"]:::external
    ONNX["ONNX Runtime"]:::external
    PT["PyTorch"]:::external
    FF["FFmpeg"]:::external

    %% ClearVoice Inference Module
    subgraph "ClearVoice (Inference)" 
        direction TB
        CL["ClearVoice Library"]:::inference
        Config["Config Loader"]:::inference
        Dataloader["Data I/O & Dataloader"]:::inference
        NetWrap["Network Wrapper & Model Factory"]:::inference
        Models["Model Implementations"]:::inference
        Utils["Utility Functions"]:::inference
    end

    %% ClearVoice Demos
    Demo1["demo.py"]:::external
    Demo2["demo_Numpy2Numpy.py"]:::external
    Demo3["demo_with_more_comments.py"]:::external
    Streamlit["streamlit_app.py"]:::external

    %% Data Stores
    SampleData["Sample Audio/Video"]:::datastore

    %% Inference Flow
    Demo1 -->|"invokes"| Config
    Demo2 -->|"invokes"| Config
    Demo3 -->|"invokes"| Config
    Streamlit -->|"invokes"| Config
    HF -->|"API calls"| Config
    Config -->|"loads settings"| Dataloader
    Dataloader -->|"reads"| SampleData
    Dataloader -->|"feeds"| NetWrap
    NetWrap -->|"instantiates"| Models
    Models -->|"uses utils"| Utils
    Models -->|"produces audio/numpy"| Output["Inference Output"]:::datastore
    Output -->|"optional input"| SSCore

    %% External Dependencies Links
    PT -->|"model execution"| NetWrap
    PT -->|"training ops"| Models
    ONNX -->|"infer engine"| NetWrap
    FF -->|"media processing"| Utils

    %% Training Module
    subgraph "Train (Training Pipelines)" 
        direction TB
        DataGen["Data Generation Scripts"]:::training
        DatasetStore["Generated Datasets"]:::datastore
        SE["Speech Enhancement Pipeline"]:::training
        SSep["Speech Separation Pipeline"]:::training
        SSR["Speech Super-Resolution Pipeline"]:::training
        TSE["Target Speaker Extraction Pipeline"]:::training
        TSEO["Online Target Speaker Extraction Pipeline"]:::training
        Solver["Solver & train.py"]:::training
        Checkpoints["Model Checkpoints"]:::datastore
    end

    DataGen -->|"generates"| DatasetStore
    DatasetStore -->|"loaded by"| SE
    DatasetStore -->|"loaded by"| SSep
    DatasetStore -->|"loaded by"| SSR
    DatasetStore -->|"loaded by"| TSE
    DatasetStore -->|"loaded by"| TSEO
    SE -->|"trains"| Solver
    SSep -->|"trains"| Solver
    SSR -->|"trains"| Solver
    TSE -->|"trains"| Solver
    TSEO -->|"trains"| Solver
    Solver -->|"saves"| Checkpoints

    %% Evaluation Module
    subgraph "SpeechScore (Evaluation)" 
        direction TB
        SS_CLI["SpeechScore CLI Demo"]:::evaluation
        SSCore["SpeechScore Core"]:::evaluation
        Metrics["Metric Implementations"]:::evaluation
        Reports["Result Logs/Reports"]:::datastore
    end

    Output -->|"evaluates"| SSCore
    SS_CLI -->|"invokes"| SSCore
    SSCore -->|"computes metrics"| Metrics
    Metrics -->|"outputs"| Reports

    %% Click Events
    click CL "https://github.com/modelscope/clearervoice-studio/tree/main/clearvoice/clearvoice/"
    click Config "https://github.com/modelscope/clearervoice-studio/tree/main/clearvoice/clearvoice/config/inference/"
    click Dataloader "https://github.com/modelscope/clearervoice-studio/blob/main/clearvoice/clearvoice/dataloader/dataloader.py"
    click NetWrap "https://github.com/modelscope/clearervoice-studio/blob/main/clearvoice/clearvoice/network_wrapper.py"
    click Models "https://github.com/modelscope/clearervoice-studio/tree/main/clearvoice/clearvoice/models/"
    click Utils "https://github.com/modelscope/clearervoice-studio/tree/main/clearvoice/clearvoice/utils/"
    click Demo1 "https://github.com/modelscope/clearervoice-studio/blob/main/clearvoice/demo.py"
    click Demo2 "https://github.com/modelscope/clearervoice-studio/blob/main/clearvoice/demo_Numpy2Numpy.py"
    click Demo3 "https://github.com/modelscope/clearervoice-studio/blob/main/clearvoice/demo_with_more_comments.py"
    click Streamlit "https://github.com/modelscope/clearervoice-studio/blob/main/clearvoice/streamlit_app.py"
    click DataGen "https://github.com/modelscope/clearervoice-studio/tree/main/train/data_generation/"
    click SE "https://github.com/modelscope/clearervoice-studio/tree/main/train/speech_enhancement/"
    click SSep "https://github.com/modelscope/clearervoice-studio/tree/main/train/speech_separation/"
    click SSR "https://github.com/modelscope/clearervoice-studio/tree/main/train/speech_super_resolution/"
    click TSE "https://github.com/modelscope/clearervoice-studio/tree/main/train/target_speaker_extraction/"
    click TSEO "https://github.com/modelscope/clearervoice-studio/tree/main/train/target_speaker_extraction_online/"
    click SSCore "https://github.com/modelscope/clearervoice-studio/blob/main/speechscore/speechscore.py"
    click SS_CLI "https://github.com/modelscope/clearervoice-studio/blob/main/speechscore/demo.py"
    click Metrics "https://github.com/modelscope/clearervoice-studio/tree/main/speechscore/scores/"

    %% Styles
    classDef inference fill:#cfe2f3,stroke:#0366d6,color:#000
    classDef training fill:#d9ead3,stroke:#6aa84f,color:#000
    classDef evaluation fill:#fce5cd,stroke:#e69138,color:#000
    classDef external fill:#d9d9d9,stroke:#7f7f7f,color:#000
    classDef datastore fill:#eeeeee,stroke:#cccccc,color:#000
```
