Training in progress, step 310

preprocessor_config.json

@@ -5,7 +5,7 @@
   "hop_length": 160,
   "mel_filters": [
     [
-      0.0,
+      -0.0,
       0.02486259490251541,
       0.0,
       0.0,

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2db0da2370cc8ac5eb9e858c9cb2c3f9ca6742fda88a9254a81831c8bca88971
+oid sha256:40def94302d63cf5ec805ccc463bcc048d769c47609011255208c65d23c301fa
 size 151098921

runs/Dec14_10-16-01_129-213-88-66/1671012968.620211/events.out.tfevents.1671012968.129-213-88-66.80950.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7bc4b0ba02400bebee01f95df25f17c58c2a196fd95ade083e1502546f1f98bb
+size 5884

runs/Dec14_10-16-01_129-213-88-66/events.out.tfevents.1671012968.129-213-88-66.80950.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2c0da368c29a30826fa13f3b91f15eaf95a457be468299c4332e6d0e32cd9bd
+size 4136

runs/Dec14_10-18-27_129-213-88-66/1671013113.7664006/events.out.tfevents.1671013113.129-213-88-66.81512.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e029656fcd3f7dcdd024f8f861f07d7c7286fd410baeacce6ed1dc19a55fa49
+size 5884

runs/Dec14_10-18-27_129-213-88-66/events.out.tfevents.1671013113.129-213-88-66.81512.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:442f4da3f922983e2d2c4aa1c0e653c986dcb0b4a8c920a116cdcaed971ce010
+size 4746

src/readme.md

@@ -0,0 +1,124 @@
+## Description
+
+Fine-tuning [OpenAI Whisper](https://github.com/openai/whisper) model for Belarusian language during 
+[Whisper fine-tuning Event](https://github.com/huggingface/community-events/tree/main/whisper-fine-tuning-event)
+hosted by HuggingFace x Lambda.
+
+The code in this repository is a modified version of code from 
+[Whisper fine-tuning Event](https://github.com/huggingface/community-events/tree/main/whisper-fine-tuning-event) repo.
+
+## Tips:
+* start with a port worwarding to monitor Tensorboard logs on local computer:
+  ```
+  ssh <remote-address> -L <local_port>:localhost:<remote_tensorboard_port>
+  ```
+* Train with redirecting output to a file using `tee`
+
+## Fine-tuning todos:
+* perform evaluation of fine-tuned model on CommonVoice test set
+* Learning rate:
+  * max learning rate is not the same as LR passed as a parameter to training script. it is actually lower.
+  * when resuming training, LR scheduling behaves incorrectly
+* check exact sizes of train, eval, test sets of CommonVoice 11
+
+## Resuming training from exising checkpoint
+When resuming training from existing checkpoint:
+* it's better to save all `checkpoint-\d+` dirs. better not to rely on data saved to `output_dir` because:
+  * not all data is saved to `output_dir`. e.g. following files are not saved to `output_dir`: 
+    `optimizer.pt`, `rng_state.pth`, `scaler.pt`, `scheduler.pt`. so can't resume training in a correct way from
+    data saved to `output_dir`
+  * when resuming training from `output_dir` as a checkpoint dir, model saved to `output_dir` can be worse than
+    previously save (need to investifate further. but such happened already)
+* learning rate gets reset if passing same parameter value to training script as in previour run.<br>
+  need to provide learning rate from the last step of previous run to continue
+  training in a correct way
+  * however even if passing learning rate from the last step, in the new run it has different value than expected
+    * probably because last checkpont was chosen incorrectly
+    * or learning rate is treated as a starting learning rate at step 0 and not on step X (where we resume).<br>
+      need to try to pass same LR that was passes as a starting LR to the very first run
+* it's unclear whether decision on saving current model
+  is made by comparing current metrics with metrics of the best checkpoint. I guess model with worse performance
+  will not overwrite best model checkpoint already exising in the output dir, but need to double check.
+* we can set `ignore_data_skip=True` Training argument not to 
+  skip data items already passed to a model - that will save time on data loads.
+    * it's unclear whether order of input items in the train set (that is shuffled) will be the same 
+      across multiple reruns - i.e. it's unclear whether sampling is the same across reruns.
+    * if the sampling is the same across reruns, `ignore_data_skip=True` will lead to same items been passed to a model
+      in current run. it's OK if previous run ended with large step value on the last epoch.
+      if not, the same elements from the same epoch will be passed to a model again.
+
+## Questions:
+* What checkpoint (best, I guess) is saved in the `output_dir`? 
+  How is it overwritten when resuming training from existing checkpoint?
+
+### Prepended tokens
+* Why are there following lines in Data Collator?
+  ```python
+    # if bos token is appended in previous tokenization step,
+    # cut bos token here as it's append later anyways
+    if (labels[:, 0] == self.decoder_start_token_id).all().cpu().item():
+        labels = labels[:, 1:]
+    ```
+* `tokenizer.bos_token_id` vs `model.config.decoder_start_token_id`.<br>
+  which one to pass to Data Collator as `decoder_start_token_id` parameter?
+* Answer:
+  * In this case, the two are equivalent. You can verify this:
+    ```python
+    print(tokenizer.bos_token_id)
+    print(model.config.decoder_start_token_id)
+    ```
+
+  * Print Output:
+    ```
+    <|startoftranscript|>
+    <|startoftranscript|>
+    ```
+
+  * Technically speaking, the decoder_start_token_id is the correct convention here. Before starting generating any tokens, we initialise the generate method with a starting token, which is the decoder_start_token_id. 
+  See: https://huggingface.co/blog/how-to-generate. The decoder_start_token_id corresponds to the initial context word sequence, and is the zero'th token generated.
+
+  * We remove this token from the encoded labels in the data collator because we always set the zero'th generated token to the decoder_start_token_id. If we leave the decoder_start_token_id as part of the label sequence, then we'll predict the decoder_start_token_id as the zero'th token, and again as the first token! Because we're always forcing it as the zero'th token, we don't need to predict it as the first token, and so we remove it from the target lables
+
+  * These tokens are not forced in the generation process, and so we don't cut them in the data collator. We need to provide them to the model as target labels so that the model can learn the correct tasks from our data
+
+  * The tokens correspond to the audio language, task (translate or transcribe) and whether to predict timestamps
+
+  * We need to tell the model what language the audio corresponds to and what task it's performing during fine-tuning. This way, it learns what audio corresponds to what language, and the difference between transcribing audio vs translating it 
+
+## Notes:
+* using CommonVoice 11 dataset in a streaming way.<br>
+  use `streaming=True` for train & validation & test.<br>
+  as an alternative, we can use `streaming=False` for validation & test sets to save time on data processing.
+  but the size of validation and test sets are unknown (need to check).
+  it's likely they are going to be large - thus pre-download of these sets might not reduce 
+  overall fine-tuning time compared to streaming mode.
+* size of train set is ~370'000 audiofiles. if using `batch_size=64`, then
+  1 epoch will have ~5782 steps. <br>
+  Because of `--eval_steps="1000"` will use `--max_steps="6000"` instead of `--max_steps="5800"`
+  to have evaluation metrics computed in the end of training.
+* if using Google Colab, need to execute  `sudo chmod -R 777 .git` inside hf repo to 
+  to set right permissions to be able to push trained models to HuggingFace Hub
+* Whispers BasicTextNormalizer splits words containing apostrophe:
+  ```python
+  > from transformers.models.whisper.english_normalizer import BasicTextNormalizer
+  > normalizer = BasicTextNormalizer()
+  > normalizer("раз'яднаць")
+  'раз яднаць'
+  ```
+* That's why `BelarusianTextNormalizer` (edited version of `BasicTextNormalizer`) was added to training script:
+  ```python
+  > from run_speech_recognition_seq2seq_streaming import BelarusianTextNormalizer
+  > normalizer_be = BelarusianTextNormalizer()
+  > normalizer_be("раз'яднаць")
+  "раз'яднаць"
+  ```
+* Need to set `use_cache` to False since we're using gradient checkpointing, and the two are incompatible
+* Default Linear scheduler is used 
+* Default Adam optimizer is used
+* To save memory (and increase either model or batch_size) can experiment with:
+    * using Adafactor instead of Adam.
+      Adam requires two optimiser params per one model param, but Adafactor uses only one.
+      > A word of caution: Adafactor is untested for fine-tuning Whisper, 
+        so we are unsure sure how Adafactor performance compares to Adam!
+    * using Adam 8bit from `bitsandbytes` module. 
+      need to provide `optim="adamw_bnb_8bit"` param to `Seq2SeqTrainingArguments`

src/requirements.txt

@@ -0,0 +1,9 @@
+torch>=1.7
+torchaudio
+git+https://github.com/huggingface/transformers
+git+https://github.com/huggingface/datasets
+librosa
+jiwer
+evaluate>=0.3.0
+more-itertools
+tensorboard

src/run.sh

@@ -1,42 +1,43 @@
-python src/run_speech_recognition_seq2seq_streaming.py \
-	--model_name_or_path="openai/whisper-small" \
-	--dataset_name="mozilla-foundation/common_voice_11_0" \
-	--dataset_config_name="be" \
-	--language="be" \
-	--train_split_name="train" \
-	--eval_split_name="validation" \
-	--model_index_name="Whisper Small Belarusian" \
-    \
-	--max_steps="6000" \
-	--output_dir="./" \
-	--per_device_train_batch_size="64" \
-	--per_device_eval_batch_size="32" \
-	--logging_steps="50" \
-	--learning_rate="1e-4" \
-	--warmup_steps="500" \
-	--evaluation_strategy="steps" \
-	--eval_steps="1000" \
-	--save_strategy="steps" \
-	--save_steps="1000" \
-	--gradient_checkpointing \
-	--fp16 \
-    \
-	--shuffle_buffer_size="500" \
-	--generation_max_length="225" \
-	--max_duration_in_seconds="30" \
-	--text_column_name="sentence" \
-	--freeze_feature_encoder="False" \
-	--report_to="tensorboard" \
-	--metric_for_best_model="wer" \
-	--greater_is_better="False" \
-	--load_best_model_at_end \
-    \
-	--do_train \
-	--do_eval \
-	--ignore_data_skip \
-	--predict_with_generate \
-	--do_normalize_eval \
-	--streaming \
-	--use_auth_token \
-	--push_to_hub \
-	--hub_model_id="ales/whisper-small-belarusian"
+python src/run_speech_recognition_seq2seq_streaming.py \
+	--model_name_or_path="openai/whisper-small" \
+	--dataset_name="mozilla-foundation/common_voice_11_0" \
+	--dataset_config_name="be" \
+	--language="be" \
+	--train_split_name="train" \
+	--eval_split_name="validation" \
+	--model_index_name="Whisper Small Belarusian" \
+    \
+	--max_steps="6000" \
+	--output_dir="./" \
+	--per_device_train_batch_size="64" \
+	--per_device_eval_batch_size="64" \
+	--logging_steps="50" \
+	--logging_first_step \
+	--learning_rate="1e-4" \
+	--warmup_steps="500" \
+	--evaluation_strategy="steps" \
+	--eval_steps="1000" \
+	--save_strategy="steps" \
+	--save_steps="1000" \
+	--gradient_checkpointing \
+	--fp16 \
+    \
+	--shuffle_buffer_size="500" \
+	--generation_max_length="225" \
+	--max_duration_in_seconds="30" \
+	--text_column_name="sentence" \
+	--freeze_feature_encoder="False" \
+	--report_to="tensorboard" \
+	--metric_for_best_model="wer" \
+	--greater_is_better="False" \
+	--load_best_model_at_end \
+    \
+	--do_train \
+	--do_eval \
+	--ignore_data_skip \
+	--predict_with_generate \
+	--do_normalize_eval \
+	--streaming \
+	--use_auth_token \
+	--push_to_hub \
+	--hub_model_id="ales/whisper-small-belarusian"

src/run_debug.sh

@@ -7,7 +7,7 @@ python src/run_speech_recognition_seq2seq_streaming.py \
 	--eval_split_name="validation" \
 	--model_index_name="Whisper Tiny Belarusian" \
 	\
-	--max_steps="300" \
+	--max_steps="500" \
 	--max_eval_samples="64" \
 	--output_dir="./" \
 	--per_device_train_batch_size="32" \

src/setup_env.sh

@@ -6,12 +6,20 @@ sudo apt-get install git-lfs
 
 sudo apt-get install tmux
 
-python3 -m venv hf_env
-source hf_env/bin/activate
-echo "source ~/hf_env/bin/activate" >> ~/.bashrc
+cd ~
+echo "executing env setup from $(pwd)"
 
-git clone https://github.com/huggingface/community-events.git
-pip install -r community-events/whisper-fine-tuning-event/requirements.txt
+python3 -m venv ~/python_venvs/hf_env
+source ~/python_venvs/hf_env/bin/activate
+echo "source ~/python_venvs/hf_env/bin/activate" >> ~/.bashrc
+
+git clone https://github.com/yks72p/whisper-finetuning-be
+pip install -r ~/whisper-finetuning-be/requirements.txt
 
 git config --global credential.helper store
 huggingface-cli login
+
+echo "env setup"
+echo "! PLEASE LOGIN INTO GIT TO BE ABLE TO PUSH TO HF HUB !"
+echo "> git config --globase user.name <user_name>"
+echo "> git config --globase user.email <user_email>"

train_20221214-101827.log

@@ -0,0 +1,228 @@
+12/14/2022 10:18:27 - WARNING - __main__ - Process rank: -1, device: cuda:0, n_gpu: 1distributed training: False, 16-bits training: True
+12/14/2022 10:18:27 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
+_n_gpu=1,
+adafactor=False,
+adam_beta1=0.9,
+adam_beta2=0.999,
+adam_epsilon=1e-08,
+auto_find_batch_size=False,
+bf16=False,
+bf16_full_eval=False,
+data_seed=None,
+dataloader_drop_last=False,
+dataloader_num_workers=0,
+dataloader_pin_memory=True,
+ddp_bucket_cap_mb=None,
+ddp_find_unused_parameters=None,
+ddp_timeout=1800,
+debug=[],
+deepspeed=None,
+disable_tqdm=False,
+do_eval=True,
+do_predict=False,
+do_train=True,
+eval_accumulation_steps=None,
+eval_delay=0,
+eval_steps=10,
+evaluation_strategy=steps,
+fp16=True,
+fp16_backend=auto,
+fp16_full_eval=False,
+fp16_opt_level=O1,
+fsdp=[],
+fsdp_min_num_params=0,
+fsdp_transformer_layer_cls_to_wrap=None,
+full_determinism=False,
+generation_max_length=225,
+generation_num_beams=None,
+gradient_accumulation_steps=1,
+gradient_checkpointing=True,
+greater_is_better=False,
+group_by_length=False,
+half_precision_backend=auto,
+hub_model_id=ales/whisper-tiny-be-test,
+hub_private_repo=False,
+hub_strategy=every_save,
+hub_token=<HUB_TOKEN>,
+ignore_data_skip=True,
+include_inputs_for_metrics=False,
+jit_mode_eval=False,
+label_names=None,
+label_smoothing_factor=0.0,
+learning_rate=0.0001,
+length_column_name=length,
+load_best_model_at_end=True,
+local_rank=-1,
+log_level=passive,
+log_level_replica=passive,
+log_on_each_node=True,
+logging_dir=./runs/Dec14_10-18-27_129-213-88-66,
+logging_first_step=True,
+logging_nan_inf_filter=True,
+logging_steps=10,
+logging_strategy=steps,
+lr_scheduler_type=linear,
+max_grad_norm=1.0,
+max_steps=500,
+metric_for_best_model=wer,
+mp_parameters=,
+no_cuda=False,
+num_train_epochs=3.0,
+optim=adamw_hf,
+optim_args=None,
+output_dir=./,
+overwrite_output_dir=False,
+past_index=-1,
+per_device_eval_batch_size=32,
+per_device_train_batch_size=32,
+predict_with_generate=True,
+prediction_loss_only=False,
+push_to_hub=True,
+push_to_hub_model_id=None,
+push_to_hub_organization=None,
+push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
+ray_scope=last,
+remove_unused_columns=True,
+report_to=['tensorboard'],
+resume_from_checkpoint=None,
+run_name=./,
+save_on_each_node=False,
+save_steps=10,
+save_strategy=steps,
+save_total_limit=None,
+seed=42,
+sharded_ddp=[],
+skip_memory_metrics=True,
+sortish_sampler=False,
+tf32=None,
+torch_compile=False,
+torch_compile_backend=None,
+torch_compile_mode=None,
+torchdynamo=None,
+tpu_metrics_debug=False,
+tpu_num_cores=None,
+use_ipex=False,
+use_legacy_prediction_loop=False,
+use_mps_device=False,
+warmup_ratio=0.0,
+warmup_steps=10,
+weight_decay=0.0,
+xpu_backend=None,
+)
+12/14/2022 10:18:27 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
+_n_gpu=1,
+adafactor=False,
+adam_beta1=0.9,
+adam_beta2=0.999,
+adam_epsilon=1e-08,
+auto_find_batch_size=False,
+bf16=False,
+bf16_full_eval=False,
+data_seed=None,
+dataloader_drop_last=False,
+dataloader_num_workers=0,
+dataloader_pin_memory=True,
+ddp_bucket_cap_mb=None,
+ddp_find_unused_parameters=None,
+ddp_timeout=1800,
+debug=[],
+deepspeed=None,
+disable_tqdm=False,
+do_eval=True,
+do_predict=False,
+do_train=True,
+eval_accumulation_steps=None,
+eval_delay=0,
+eval_steps=10,
+evaluation_strategy=steps,
+fp16=True,
+fp16_backend=auto,
+fp16_full_eval=False,
+fp16_opt_level=O1,
+fsdp=[],
+fsdp_min_num_params=0,
+fsdp_transformer_layer_cls_to_wrap=None,
+full_determinism=False,
+generation_max_length=225,
+generation_num_beams=None,
+gradient_accumulation_steps=1,
+gradient_checkpointing=True,
+greater_is_better=False,
+group_by_length=False,
+half_precision_backend=auto,
+hub_model_id=ales/whisper-tiny-be-test,
+hub_private_repo=False,
+hub_strategy=every_save,
+hub_token=<HUB_TOKEN>,
+ignore_data_skip=True,
+include_inputs_for_metrics=False,
+jit_mode_eval=False,
+label_names=None,
+label_smoothing_factor=0.0,
+learning_rate=0.0001,
+length_column_name=length,
+load_best_model_at_end=True,
+local_rank=-1,
+log_level=passive,
+log_level_replica=passive,
+log_on_each_node=True,
+logging_dir=./runs/Dec14_10-18-27_129-213-88-66,
+logging_first_step=True,
+logging_nan_inf_filter=True,
+logging_steps=10,
+logging_strategy=steps,
+lr_scheduler_type=linear,
+max_grad_norm=1.0,
+max_steps=500,
+metric_for_best_model=wer,
+mp_parameters=,
+no_cuda=False,
+num_train_epochs=3.0,
+optim=adamw_hf,
+optim_args=None,
+output_dir=./,
+overwrite_output_dir=False,
+past_index=-1,
+per_device_eval_batch_size=32,
+per_device_train_batch_size=32,
+predict_with_generate=True,
+prediction_loss_only=False,
+push_to_hub=True,
+push_to_hub_model_id=None,
+push_to_hub_organization=None,
+push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
+ray_scope=last,
+remove_unused_columns=True,
+report_to=['tensorboard'],
+resume_from_checkpoint=None,
+run_name=./,
+save_on_each_node=False,
+save_steps=10,
+save_strategy=steps,
+save_total_limit=None,
+seed=42,
+sharded_ddp=[],
+skip_memory_metrics=True,
+sortish_sampler=False,
+tf32=None,
+torch_compile=False,
+torch_compile_backend=None,
+torch_compile_mode=None,
+torchdynamo=None,
+tpu_metrics_debug=False,
+tpu_num_cores=None,
+use_ipex=False,
+use_legacy_prediction_loop=False,
+use_mps_device=False,
+warmup_ratio=0.0,
+warmup_steps=10,
+weight_decay=0.0,
+xpu_backend=None,
+)
+12/14/2022 10:18:27 - INFO - __main__ - output_dir already exists. will try to load last checkpoint.
+12/14/2022 10:18:27 - INFO - __main__ - last_checkpoint is None. will try to read from training_args.resume_from_checkpoint
+12/14/2022 10:18:27 - INFO - __main__ - last_checkpoint is None. resume_from_checkpoint is either None or not existing dir. will try to read from the model saved in the root of output_dir.
+12/14/2022 10:18:27 - INFO - __main__ - found pytorch_model.bin inside output_dir. will continue training treating output_dir as a last checkpoint.
+12/14/2022 10:18:28 - INFO - datasets.info - Loading Dataset Infos from /home/ubuntu/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_11_0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f
+12/14/2022 10:18:28 - INFO - datasets.info - Loading Dataset Infos from /home/ubuntu/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_11_0/f8e47235d9b4e68fa24ed71d63266a02018ccf7194b2a8c9c598a5f3ab304d9f
+12/14/2022 10:18:31 - WARNING - huggingface_hub.repository - /home/ubuntu/whisper-tiny-be-test/./ is already a clone of https://huggingface.co/ales/whisper-tiny-be-test. Make sure you pull the latest changes with `repo.git_pull()`.

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:350c7e60ed35dd7fe825968ffb77600485c958e44f8b775cc30447b1f2b880aa
+oid sha256:1437f82ab7eeff36134dc9d16b104ff12b88d24a6de7414f4fa0a3a341e9c3fd
 size 3643