kawshikbuet17/orpheus-tts-bengali-finetuning

1. Repo purpose and top-level split

- This repository is organized around one core product: a text-to-speech model family called Orpheus.

- The codebase is split into three main concerns: model runtime/inference, training (pretrain + finetune), and deployment/examples.

2. Core inference package boundary

- The packaged runtime lives in orpheus_tts_pypi/orpheus_tts.

- engine_class.py and decoder.py appear to define the public model interface and the generation/decoding path.

- This is the boundary most users hit when they import OrpheusModel and call speech generation.

3. Published package vs local repository code

- The repo includes a PyPI-packaged distribution (orpheus_tts_pypi) with packaging metadata in pyproject.toml and setup.py.

- The README explicitly notes that some users may need to import from the local package copy instead of the published orpheus-speech package when fixes have not yet been released.

- For architecture diagrams, treat this as a distribution boundary: source repo code vs installed package.

4. Inference data flow

- The primary inference flow is: text prompt + voice selection + generation parameters → streamed audio chunks → WAV/file/client playback.

- The README’s example writes mono 24 kHz PCM audio incrementally as chunks arrive, which indicates the model generates streaming audio tokens/frames, not a single blocking output.

- This streaming behavior is central to the low-latency claim and is likely the main runtime path in the package.

5. Model hosting and external inference options

- The repo includes an official Baseten example under additional_inference_options/baseten_inference_example, indicating a production deployment path outside the local Python package.

- That path is a separate inference boundary: client code → remote hosted Orpheus endpoint → streaming audio response.

- The README treats Baseten as the preferred production inference partner.

6. No-GPU inference alternative

- additional_inference_options/no_gpu provides a CPU/LLM.cpp-style fallback path for running without CUDA/GPU.

- Architecturally, this is an alternative runtime backend rather than a separate product: same TTS task, different execution engine and hardware assumptions.

7. Audio post-processing / watermarking

- additional_inference_options/watermark_audio adds an optional audio post-processing stage called Silent Cipher watermarking.

- This is not part of the core model; it is a downstream audio transform applied after generation, so it belongs as an optional post-inference processing branch.

8. Realtime streaming demo

- realtime_streaming_example contains a Python server (main.py) and browser client (client.html).

- This forms a small end-to-end streaming system: browser UI → local streaming server → Orpheus inference → live audio playback in the browser.

- It demonstrates the model’s low-latency behavior and the integration point between model output and client consumption.

9. Training subsystem: pretraining

- pretrain/ contains a standalone training pipeline for the base model, with its own config.yaml, readme.md, and train.py.

- This path is for large-scale model training from broader text/speech corpora and is conceptually separate from speaker-specific finetuning.

- The README describes it as analogous to LLM training with long sequences and chained inputs.

10. Training subsystem: finetuning

- finetune/ contains the finetuning pipeline, including train.py, lora.py, and multiple configs for standard and Bengali variants.

- The directory also includes scripts for full training and QLoRA-based training, showing that finetuning supports multiple adaptation strategies.

- The likely flow is: prepared HF dataset → config-driven training script → LoRA/QLoRA or full finetune → checkpoint.

11. Data preparation / preprocessing

- preprocess_bengali_telecom_orpheus_local.py is a data preparation script for a specific Bengali telecom dataset flow.

- This indicates a preprocessing boundary between raw audio/text sources and the training dataset format consumed by the finetuning pipeline.

- The README also references Hugging Face datasets as the canonical training input shape.

12. Evaluation and testing

- test_orpheus_repo.py and test_orpheus_concurrent.py provide validation around repository behavior and concurrency, which is especially relevant for streaming generation.

- orpheus_benchmark_ui.py suggests an internal benchmarking interface for comparing latency/throughput or generation quality across configurations.

- These are supporting systems around the inference path rather than part of the core model.

13. Configuration and environment

- Root-level requirements.txt, requirements_finetune.txt, .python-version, and multiple YAML configs show that the repo is managed as a Python-first project with separate runtime and training dependency sets.

- The README references transformers, datasets, wandb, trl, flash_attn, torch, accelerate, and vllm, which materially shape the architecture: training is Hugging Face/Accelerate-based, and fast inference is vLLM-backed.

14. What to show in an architecture diagram

- A clear diagram should show four layers: clients/examples, model runtime package, training pipelines, and optional deployment/post-processing integrations.

- The most important arrows are: prompt/dataset input → model generation/training → streamed audio/checkpoint output, with separate branches for Baseten hosting, CPU fallback, and watermarking.