How aiola-lab/whisper-ner works
Project type and purpose
• WhisperNER is an open-source Python package implementing the “WhisperNER” research paper, a unified model for automatic speech recognition (ASR) and named entity recognition (NER).
• It provides components for dataset loading, model definition (based on OpenAI’s Whisper), training/finetuning pipelines, and inference utilities with zero-shot NER capabilities.
File structure
• Top-level files:
• setup.py / requirements.txt → package installation and dependencies (PyTorch, torchaudio, Hugging Face, wandb, etc.)
• Package directory: whisper_ner/
• dataset.py → reads JSON records (text, audio, ner annotations) and constructs PyTorch Dataset/DataLoader
• models.py → wraps or extends the base Whisper model, integrates NER head, handles prompt conditioning
• trainer.py → orchestrates training/finetuning (argument parsing, optimizer, scheduler, gradient accumulation, checkpointing, wandb logging)
• inference.py → loads a pretrained model, processes an audio file and a prompt, applies entity_logits_processor, outputs transcription + entities
• utils/ → reusable utilities:
– entity_logits_processor.py → post-processing to extract NER spans from model logits
– metrics.py → computation of NER metrics (precision/recall/F1) and ASR metrics (WER)
– utils.py → shared helper functions (e.g., loading audio, tokenization, prompt encoding)
README for additional insights
• Confirms the architecture is research-oriented: ASR+NER unified model
• Highlights external integrations:
• huggingface model Hub (aiola/whisper-ner-v1)
• wandb for experiment tracking
• support for LoRA fine-tuning
• Shows example dataset JSON format and CLI commands for trainer.py and inference.py