How vrclc-duk/st-ta-en works
Project Type and Purpose
• This is an end-to-end speech translation system using the Whisper model. Its purpose is twofold—training (fine-tuning the Whisper model on Tamil speech data) and inference (translating Tamil speech into English text with associated quality metrics).
Analyzing the File Structure
• The project has a lean file structure with key scripts: one for training (“ST_training.py” as per the file tree, mentioned as ST-train.py in the README) and one for inference (“ST-inference.py”), plus a README and a requirements file.
• This structure indicates that the project is designed around two primary pipelines which share common dependencies (found in “requirements.txt”).
Main Components
a. Training Pipeline:
• Input Data: Tamil speech data (audio waveforms) provided for training.
• Preprocessing/Validation: Implicit component where audio and text data may be prepared and validated.
• Whisper Model Fine-Tuning: Core component in “ST_training.py” that adjusts the pre-trained Whisper model on Tamil data.
• Checkpoint & Log Management: Saving model checkpoints and tracking training metrics.
• Evaluation: Compute BLEU and METEOR scores to gauge training performance.
b. Inference Pipeline:
• Model Loading: Retrieval of the trained or fine-tuned Whisper model from either the HF Hub or local checkpoints.
• Audio Processing: Module for accepting and processing incoming Tamil audio (batch processing support as described).
• Translation Generation: Translating the processed audio into English text.
• Quality Metrics Computation: Using SacreBLEU and METEOR to measure translation quality.
• Output: Writing results to a CSV containing original text (if available), ground truth, model-generated translation, and metric scores.
c. Shared Components:
• Dependencies and Libraries: Essential packages (PyTorch, Transformers, Datasets, SacreBLEU, METEOR, NumPy, pandas) that support both training and inference.
• Configuration: Environment settings and dependency management specified in “requirements.txt”.
Relationships & Interactions
• The training component takes in raw data and outputs a fine-tuned Whisper model (plus logs and checkpoints), which then acts as the internal “product” of the training process.
• The inference component depends on loading the trained model, processing new audio input, running the translation, and then evaluating the result.
• From raw data → Training pipeline → Model checkpoints/outputs.
• From Model checkpoints → Inference pipeline.
• From inference → Processed audio → output CSV and metrics.
Architectural Patterns & Design Principles
• Separation of Concerns: Clear division between training and inference, each handling specific aspects of the model lifecycle.
• Pipeline Structure: A sequential flow where training feeds “model artifacts” to inference.
• Modularity: Although the project is small-scale, think in terms of modular components (data handling, model management, evaluation) that can expand if needed.
Including Relevant Technologies
• Use of deep learning frameworks such as PyTorch for model operations.
• Significance of Hugging Face’s Transformers library which is central for implementing the Whisper model.
• Evaluation tools (SacreBLEU, METEOR) as critical for translation assessment.