How pr0mila/medibeng-whisper-tiny works
Project type and purpose
• This is an ML-centric application combining:
• A training/fine-tuning pipeline for a speech translation model (Whisper-Tiny → MediBeng-Whisper-Tiny).
• An inference service exposing the fine-tuned model via both a Gradio UI and a FastAPI REST endpoint.
• Purpose: ingest code-switched Bengali-English audio, translate to English, and return text (for clinical settings).
File structure
• Top-level scripts:
• main.py – orchestrates fine-tuning
• data_loader.py – loads and preprocesses dataset
• upload_model/upload_model.py – pushes model to Hugging Face Hub
• Config:
• config/config.py – hyperparameters, model names, output paths
• Inference app:
• app/api.py – FastAPI server
• app/gradio_interface.py – Gradio web UI
• app/audio_processing.py – audio feature extraction and post-processing
• Assets, tests, requirements define static resources, test cases, and dependencies.
Key README insights
• Two interaction modes: Gradio UI and FastAPI endpoint (/transcribe).
• Training uses Hugging Face Transformers, Accelerate, datasets → outputs WER metrics.
• Model and dataset live on Hugging Face Hub; upload script uses HF token.
Main components
a. Training Pipeline
1. Data Source:
• Local synthetic MediBeng dataset (data_loader.py)
• Hugging Face dataset (pr0mila-gh0sh/MediBeng)
2. Preprocessing:
• librosa, audio_feature_extraction (audio_processing.py or inside data_loader)
3. Model Fine-tuning:
• openai/whisper-tiny base model → huggingface/transformers Trainer via main.py
• Config parameters from config/config.py
4. Output:
• Saved model checkpoints in OUTPUT_DIR
5. Upload:
• Hugging Face Hub via upload_model/upload_model.py
b. Inference Stack
1. Client Layer:
• Gradio Web UI (app/gradio_interface.py)
• External REST client (cURL or browser hitting FastAPI endpoint)
2. API Layer:
• FastAPI server (app/api.py) exposes /transcribe
• Both UI and API import audio_processing.py
3. Preprocessing & Postprocessing:
• Audio file ingestion → waveform extraction → feature normalization
• Whisper model inference → raw logits → decode to English text
4. Model Serving:
• Load MediBeng-Whisper-Tiny weights from local or HF Hub
• Hugging Face Transformers/Accelerate runtime
5. Response:
• Return translated text payload in JSON (API) or render in the web UI (Gradio)
c. External Services & Storage
• Hugging Face Hub (model & dataset)
• CTranslate2 conversion (optional, per README)
• Local filesystem (assets/, tests/data/, OUTPUT_DIR)
Relationships and interactions
• Training → pushes model to HF Hub → inference layers pull model
• Clients → UI/API → Preprocessing → Model → Postprocessing → Client
• Config drives both training and inference parameters
Architectural patterns & principles
• Clear separation of concerns:
• Data ingestion (data_loader.py)
• Model logic (main.py, upload_model.py)
• Inference API (app/api.py)
• Presentation/UI (app/gradio_interface.py)
• Modular design to swap out base models or interfaces
• Use of external managed services (Hugging Face Hub) for model storage and retrieval
Elements
• A title indicating “MediBeng-Whisper-Tiny System Architecture”