# aiola-lab/whisper-ner architecture

> Architecture diagram of the GitHub repository aiola-lab/whisper-ner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aiola-lab/whisper-ner
- Repository: https://github.com/aiola-lab/whisper-ner

## Overview

##### Project type and purpose
- WhisperNER is an open-source Python package implementing the “WhisperNER” research paper, a unified model for automatic speech recognition (ASR) and named entity recognition (NER).
- It provides components for dataset loading, model definition (based on OpenAI’s Whisper), training/finetuning pipelines, and inference utilities with zero-shot NER capabilities.

##### File structure
- Top-level files:
- setup.py / requirements.txt → package installation and dependencies (PyTorch, torchaudio, Hugging Face, wandb, etc.)
- Package directory: whisper_ner/
- dataset.py → reads JSON records (text, audio, ner annotations) and constructs PyTorch Dataset/DataLoader
- models.py → wraps or extends the base Whisper model, integrates NER head, handles prompt conditioning
- trainer.py → orchestrates training/finetuning (argument parsing, optimizer, scheduler, gradient accumulation, checkpointing, wandb logging)
- inference.py → loads a pretrained model, processes an audio file and a prompt, applies entity_logits_processor, outputs transcription + entities
- utils/ → reusable utilities:
– entity_logits_processor.py → post-processing to extract NER spans from model logits
– metrics.py → computation of NER metrics (precision/recall/F1) and ASR metrics (WER)
– utils.py → shared helper functions (e.g., loading audio, tokenization, prompt encoding)

##### README for additional insights
- Confirms the architecture is research-oriented: ASR+NER unified model
- Highlights external integrations:
- huggingface model Hub (aiola/whisper-ner-v1)
- wandb for experiment tracking
- support for LoRA fine-tuning
- Shows example dataset JSON format and CLI commands for trainer.py and inference.py

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    JSON["JSON Dataset"]:::data
    Audio["Audio Files"]:::data
    JSON -->|reads| Dataset
    Audio -->|reads via torchaudio| Dataset
    Dataset["Dataset Loader"]:::data
    click Dataset "https://github.com/aiola-lab/whisper-ner/blob/main/whisper_ner/dataset.py"
    Dataset -->|creates| DataLoader["PyTorch DataLoader"]:::data

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        Encoder["Whisper Encoder"]:::model
        Head["NER Head"]:::model
    end
    click Encoder "https://github.com/aiola-lab/whisper-ner/blob/main/whisper_ner/models.py"
    click Head "https://github.com/aiola-lab/whisper-ner/blob/main/whisper_ner/models.py"
    DataLoader -->|batches→tokens & audio| Encoder
    Encoder -->|encodings| Head

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        Trainer["Trainer"]:::train
        Loss["Loss Computation"]:::train
        Optimizer["Optimizer & Scheduler"]:::train
        CheckpointStore["Checkpoint\n(cylinder)"]:::storage
    end
    click Trainer "https://github.com/aiola-lab/whisper-ner/blob/main/whisper_ner/trainer.py"
    Trainer -->|fetches| DataLoader
    Trainer -->|calls forward| Encoder
    Head -->|logits| Loss
    Loss -->|backprop| Optimizer
    Optimizer -->|updates| Encoder
    Optimizer -->|updates| Head
    Trainer -->|saves| CheckpointStore
    Trainer -->|logs| wandb["wandb"]:::external
    Trainer -->|uploads| HF["Hugging Face Hub"]:::external

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        direction TB
        InferenceModule["Inference Module"]:::infer
        Preprocess["Pre-processing"]:::infer
        Forward["Forward Pass"]:::infer
        Postproc["Post-process logits"]:::infer
        Output["Transcription + Entities"]:::infer
    end
    click InferenceModule "https://github.com/aiola-lab/whisper-ner/blob/main/whisper_ner/inference.py"
    Audio -->|load| Preprocess
    Preprocess -->|tokens & audio| InferenceModule
    InferenceModule -->|forward| Forward
    Forward -->|raw logits| Postproc
    click Postproc "https://github.com/aiola-lab/whisper-ner/blob/main/whisper_ner/utils/entity_logits_processor.py"
    Postproc -->|entities| Output

    %% Utilities
    subgraph "Utilities"
        direction TB
        EntityProc["EntityLogitsProcessor"]:::util
        Metrics["Metrics Calculator"]:::util
        Utils["Helper Utilities"]:::util
    end
    click EntityProc "https://github.com/aiola-lab/whisper-ner/blob/main/whisper_ner/utils/entity_logits_processor.py"
    click Metrics "https://github.com/aiola-lab/whisper-ner/blob/main/whisper_ner/utils/metrics.py"
    click Utils "https://github.com/aiola-lab/whisper-ner/blob/main/whisper_ner/utils/utils.py"
    Head -->|logits| EntityProc
    Loss -->|metrics| Metrics
    Output -->|evaluate| Metrics
    EntityProc --> Metrics

    %% Installation & Docs
    Setup["setup.py / requirements.txt"]:::infra
    click Setup "https://github.com/aiola-lab/whisper-ner/blob/main/setup.py"
    README["README.md"]:::infra
    Assets["assets/"]:::infra
    README --> Assets

    %% Styles
    classDef data fill:#bbdefb,stroke:#333
    classDef model fill:#c8e6c9,stroke:#333
    classDef train fill:#ffe0b2,stroke:#333
    classDef infer fill:#e1bee7,stroke:#333
    classDef util fill:#f0f4c3,stroke:#333
    classDef external fill:#eeeeee,stroke:#333,stroke-dasharray: 4 2
    classDef storage fill:#b3e5fc,stroke:#333,stroke-dasharray: 2 2
    classDef infra fill:#d7ccc8,stroke:#333,stroke-dasharray: 5 5
```
