README.md

---
language:
- nl
license: apache-2.0
base_model: openai/whisper-medium
tags:
- nl-asr-leaderboard
- generated_from_trainer
- noise
datasets:
- mozilla-foundation/common_voice_11_0
metrics:
- wer
model-index:
- name: Whisper Medium NL - Noise
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Common Voice 11.0
      type: mozilla-foundation/common_voice_11_0
      config: nl
      split: test[:250]
      args: 'config: nl, split: test'
    metrics:
    - name: Wer
      type: wer
      value: 22.04155374887082
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# whisper-nl-noise

This model is a fine-tuned version of [openai/whisper-medium](https://huggingface.co/openai/whisper-medium) on the the (dutch) [mozilla/common-voice](https://commonvoice.mozilla.org/en/datasets) dataset (11.0). This dataset is augmented with various forms of background noise, retrieved from [pixabay](https://pixabay.com/sound-effects/search/car/) and outtakes from specific audio tracks.


## Model description

### Whisper (base)
Whisper is a pre-trained model for automatic speech recognition (ASR) and speech translation. Trained on 680k hours of labelled data, Whisper models demonstrate a strong ability to generalize to many datasets and domains without the need for fine-tuning.

Whisper was proposed in the paper [Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356) by Alec Radford et al from OpenAI. The original code repository can be found [here](https://github.com/openai/whisper).

### Whisper NL (noise)
This current Whisper model is specifically fine-tuned on noisy Dutch data. It intends to have increased performance on this task. The generalizing aspects of the model will however be lost in this process. It will nevertheless build upon the 680 hours of labeled data the base model already received during training. 

## Intended uses & limitations

This model is specifically trained for (very) noise (Dutch) audio. It is expected that is performs worse on audio files which do not meet these criteria. 

Noise is often bound to specific contexts and recordings. The model will therefore not generalize to all sorts and types of (car) noise.

## Training and evaluation data

The [mozilla/common-voice](https://commonvoice.mozilla.org/en/datasets) dataset (11.0) was used. With the predefined 'train' and 'test' split. For reasons of time-management, only the first 5% of the test set was used.

## Training procedure
The training procedure as outlined in the original [Huggingface blog](https://huggingface.co/blog/fine-tune-whisper) was used, see [here](https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/fine_tune_whisper.ipynb).

The only alteration was made during preparation of the dataset (`prepare_dataset(batch)`). Within this step the data was [augmented](https://pytorch.org/audio/stable/tutorials/audio_data_augmentation_tutorial.html) to include various samples of background noise. These noises were sampled from various audio files and sources. For each audio track used from the common-voice dataset, a random noise sample was mixed with this audio track. The signal to noise ratio (SNR) varied (randomly) between -5, and 1 dB.

$$ \mathrm{SNR} = {{P_{signal}} \over {P_{noise}}} $$

$$ \mathrm{SNR_{dB}} = 10 \log _{10}\mathrm{SNR} $$

This SNR range makes the noise disturbance, on average, quite invasive.
The augmentation, does, however not alter the transcription of the audio track, these remain unchanged. 

Whilst it would be beneficial to add the noise on the fly during training, to enable varying selections of noise for the same audio track, for efficiency reasons this strategy was not applied. Each audio tracks is therefore augmented with a single (randomly selected) noise track. The length of the audio track remained unchanged. If needed the noise track was repeated or truncated to meet the length of the audio track.

### Training hyperparameters

<details>
<summary> Hyperparameters </summary>
```txt
- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- training_steps: 10000
- mixed_precision_training: Native AMP
```
</details>

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Wer     |
|:-------------:|:-----:|:-----:|:---------------:|:-------:|
| 0.1974        | 0.97  | 5000  | 0.4473          | 25.7904 |
| 0.0668        | 1.95  | 10000 | 0.3982          | 22.0416 |



### Framework versions

- Transformers 4.37.2
- Pytorch 2.1.0+cu121
- Datasets 2.17.1
- Tokenizers 0.15.2