# cikay/voice_dataset_pipeline architecture

> Architecture diagram of the GitHub repository cikay/voice_dataset_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cikay/voice_dataset_pipeline
- Repository: https://github.com/cikay/voice_dataset_pipeline

## Overview

This repository prepares paired audio/text data for speech-model training. The principal workflow is configured stage orchestration: enhance source audio, align text to audio and create utterance segments, filter candidates, then cluster segments by speaker; optional scoring and statistics stages enrich or summarize results. Manifests carry metadata between stages. A separate publishing command loads a manifest into a Hugging Face dataset and pushes it to the Hub. The README documents these capabilities, while some stage implementations were not sampled, so their internal wiring is not inferred.

Main entry points: [`voice_dataset_pipeline/pipeline.py`](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/pipeline.py) (Stage runner), [`voice_dataset_pipeline/enhancement/stage.py`](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/enhancement/stage.py) (Audio enhancement), [`voice_dataset_pipeline/segmentation/stage.py`](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/segmentation/stage.py) (Forced alignment segments), [`voice_dataset_pipeline/push_dataset.py`](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/push_dataset.py) (Publish command)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline orchestration"]
  node_pipeline["Stage runner<br/>[pipeline.py]"]
  node_stage_base["Manifest utilities<br/>[stage.py]"]
  node_manifests["Stage manifests"]
end

subgraph group_processing["Audio processing"]
  node_enhancement["Audio enhancement<br/>[stage.py]"]
  node_deepfilter["DeepFilterNet<br/>[deepfilternet.py]"]
  node_segmentation["Forced alignment segments<br/>[stage.py]"]
end

subgraph group_curation["Curation and analysis"]
  node_filter["Segment filtering<br/>[stage.py]"]
  node_quality["Quality scoring<br/>[stage.py]"]
  node_speaker["Speaker clustering<br/>[stage.py]"]
  node_stats["Dataset statistics<br/>[stage.py]"]
end

subgraph group_publishing["Dataset publishing"]
  node_hub_publish["Publish command<br/>[push_dataset.py]"]
  node_local_loader["Local dataset loader"]
end

node_operator(("Dataset operator"))
node_source_data["Paired source data"]
node_aligner["CTC aligner"]
node_hub["Hugging Face Hub"]

node_operator -->|"starts"| node_pipeline
node_pipeline -->|"dispatches"| node_enhancement
node_pipeline -->|"dispatches"| node_segmentation
node_pipeline -->|"dispatches"| node_filter
node_pipeline -->|"dispatches"| node_quality
node_pipeline -->|"dispatches"| node_speaker
node_pipeline -->|"dispatches"| node_stats
node_pipeline -->|"uses utilities"| node_stage_base
node_source_data -->|"audio and manifest"| node_enhancement
node_enhancement -.->|"invokes tool"| node_deepfilter
node_enhancement -->|"writes enhanced manifest"| node_manifests
node_manifests -->|"input manifest"| node_segmentation
node_segmentation -->|"aligns text and audio"| node_aligner
node_segmentation -->|"writes segments"| node_manifests
node_manifests -->|"input manifest"| node_filter
node_filter -->|"writes filtered manifest"| node_manifests
node_manifests -->|"input manifest"| node_quality
node_quality -->|"writes scores"| node_manifests
node_manifests -->|"input manifest"| node_speaker
node_speaker -->|"writes speaker IDs"| node_manifests
node_manifests -->|"input manifest"| node_stats
node_stats -->|"writes statistics"| node_manifests
node_operator -->|"publishes dataset"| node_hub_publish
node_hub_publish -->|"loads manifest"| node_local_loader
node_hub_publish -->|"pushes dataset"| node_hub
node_manifests -->|"speaker manifest"| node_hub_publish

click node_pipeline "https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/pipeline.py"
click node_stage_base "https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/stage.py"
click node_enhancement "https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/enhancement/stage.py"
click node_deepfilter "https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/enhancement/tools/deepfilternet.py"
click node_segmentation "https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/segmentation/stage.py"
click node_filter "https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/filter/stage.py"
click node_quality "https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/quality_score/stage.py"
click node_speaker "https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/speaker_clustering/stage.py"
click node_stats "https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/stats/stage.py"
click node_hub_publish "https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/push_dataset.py"
click node_local_loader "https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/load_dataset_from_local.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_stage_base,node_manifests toneBlue
class node_enhancement,node_deepfilter,node_segmentation toneAmber
class node_filter,node_quality,node_speaker,node_stats toneMint
class node_hub_publish,node_local_loader toneRose
class node_operator,node_source_data,node_aligner,node_hub toneIndigo
```

## Components

### Pipeline orchestration

- [Stage runner](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/pipeline.py): `voice_dataset_pipeline/pipeline.py`
- [Manifest utilities](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/stage.py): `voice_dataset_pipeline/stage.py`
- **Stage manifests**

### Audio processing

- [Audio enhancement](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/enhancement/stage.py): `voice_dataset_pipeline/enhancement/stage.py`
- [DeepFilterNet](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/enhancement/tools/deepfilternet.py): `voice_dataset_pipeline/enhancement/tools/deepfilternet.py`
- [Forced alignment segments](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/segmentation/stage.py): `voice_dataset_pipeline/segmentation/stage.py`

### Curation and analysis

- [Segment filtering](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/filter/stage.py): `voice_dataset_pipeline/filter/stage.py`
- [Quality scoring](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/quality_score/stage.py): `voice_dataset_pipeline/quality_score/stage.py`
- [Speaker clustering](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/speaker_clustering/stage.py): `voice_dataset_pipeline/speaker_clustering/stage.py`
- [Dataset statistics](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/stats/stage.py): `voice_dataset_pipeline/stats/stage.py`

### Dataset publishing

- [Publish command](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/push_dataset.py): `voice_dataset_pipeline/push_dataset.py`
- [Local dataset loader](https://github.com/cikay/voice_dataset_pipeline/blob/main/voice_dataset_pipeline/load_dataset_from_local.py): `voice_dataset_pipeline/load_dataset_from_local.py`

### Other components

- **Dataset operator**
- **Paired source data**
- **CTC aligner**
- **Hugging Face Hub**

## Connections

- Dataset operator → Stage runner: starts
- Stage runner → Audio enhancement: dispatches
- Stage runner → Forced alignment segments: dispatches
- Stage runner → Segment filtering: dispatches
- Stage runner → Quality scoring: dispatches
- Stage runner → Speaker clustering: dispatches
- Stage runner → Dataset statistics: dispatches
- Stage runner → Manifest utilities: uses utilities
- Paired source data → Audio enhancement: audio and manifest
- Audio enhancement → DeepFilterNet: invokes tool
- Audio enhancement → Stage manifests: writes enhanced manifest
- Stage manifests → Forced alignment segments: input manifest
- Forced alignment segments → CTC aligner: aligns text and audio
- Forced alignment segments → Stage manifests: writes segments
- Stage manifests → Segment filtering: input manifest
- Segment filtering → Stage manifests: writes filtered manifest
- Stage manifests → Quality scoring: input manifest
- Quality scoring → Stage manifests: writes scores
- Stage manifests → Speaker clustering: input manifest
- Speaker clustering → Stage manifests: writes speaker IDs
- Stage manifests → Dataset statistics: input manifest
- Dataset statistics → Stage manifests: writes statistics
- Dataset operator → Publish command: publishes dataset
- Publish command → Local dataset loader: loads manifest
- Publish command → Hugging Face Hub: pushes dataset
- Stage manifests → Publish command: speaker manifest