End of training

README.md

@@ -44,32 +44,32 @@ More information needed
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
 | **teacher eval** |  | 43.25 | 61.25 |  |  |  |  | 11.6875 | 19.125 |
-| 0 | 0 | 1829656068096.0 | 79714593013760.0 | 26.8020 | 30.2396 | 82.673 | 10.351 | 4898947072.0 | 21715354648576.0 |
-| 2500 | 0.0404 | 908.0 | 7264.0 | 6.0454 | 30.2047 | 82.769 | 10.363 | 608.0 | 13888.0 |
-| 5000 | 0.0808 | 376.0 | 1888.0 | 4.9904 | 30.154 | 82.908 | 10.38 | 294.0 | 250.0 |
-| 7500 | 0.1212 | 227.0 | 904.0 | 4.4958 | 30.2026 | 82.774 | 10.363 | 199.0 | 183.0 |
-| 10000 | 0.1616 | 170.0 | 692.0 | 4.2057 | 30.1547 | 82.906 | 10.38 | 160.0 | 172.0 |
-| 12500 | 0.2020 | 129.0 | 458.0 | 3.8235 | 30.3035 | 82.499 | 10.329 | 107.0 | 127.0 |
-| 15000 | 0.2424 | 108.5 | 404.0 | 3.6530 | 30.2712 | 82.587 | 10.34 | 88.0 | 179.0 |
-| 17500 | 0.2828 | 94.0 | 362.0 | 3.5036 | 30.2085 | 82.758 | 10.361 | 75.0 | 125.0 |
-| 20000 | 0.3232 | 77.5 | 264.0 | 3.3187 | 30.1387 | 82.95 | 10.385 | 62.0 | 128.0 |
-| 22500 | 0.3636 | 66.0 | 219.0 | 3.1312 | 30.1887 | 82.812 | 10.368 | 51.0 | 95.0 |
-| 25000 | 0.4040 | 63.75 | 191.0 | 3.0365 | 30.175 | 82.85 | 10.373 | 47.0 | 69.5 |
-| 27500 | 0.4444 | 60.25 | 193.0 | 2.9840 | 30.2256 | 82.711 | 10.355 | 41.5 | 63.75 |
-| 30000 | 0.4848 | 61.0 | 196.0 | 2.9776 | 30.1558 | 82.903 | 10.379 | 41.75 | 132.0 |
-| 32500 | 0.5253 | 58.5 | 189.0 | 2.9520 | 30.1884 | 82.813 | 10.368 | 42.25 | 121.5 |
-| 35000 | 0.5657 | 56.5 | 175.0 | 2.8863 | 30.1265 | 82.983 | 10.39 | 38.5 | 59.75 |
-| 37500 | 0.6061 | 57.0 | 168.0 | 2.8672 | 30.1704 | 82.863 | 10.374 | 37.75 | 72.0 |
-| 40000 | 0.6465 | 55.0 | 173.0 | 2.8404 | 30.1803 | 82.836 | 10.371 | 35.75 | 45.75 |
-| 42500 | 0.6869 | 55.0 | 160.0 | 2.8200 | 30.1856 | 82.821 | 10.369 | 36.0 | 64.5 |
-| 45000 | 0.7273 | 50.0 | 136.0 | 2.7104 | 30.1734 | 82.854 | 10.373 | 31.25 | 42.75 |
-| 47500 | 0.7677 | 49.5 | 129.0 | 2.6868 | 30.192 | 82.803 | 10.367 | 29.75 | 37.0 |
-| 50000 | 0.8081 | 50.0 | 131.0 | 2.6746 | 30.1592 | 82.893 | 10.378 | 29.5 | 39.0 |
-| 52500 | 0.8485 | 49.5 | 126.0 | 2.6637 | 30.1577 | 82.898 | 10.379 | 28.75 | 39.25 |
-| 55000 | 0.8889 | 48.5 | 121.5 | 2.6448 | 30.156 | 82.902 | 10.379 | 28.375 | 35.5 |
-| 57500 | 0.9293 | 48.25 | 120.5 | 2.6397 | 30.2088 | 82.757 | 10.361 | 28.125 | 33.5 |
-| 60000 | 0.9697 | 48.0 | 121.0 | 2.6370 | 30.1788 | 82.84 | 10.372 | 28.0 | 33.5 |
-| 61875 | 1.0 | 48.0 | 121.0 | 2.6364 | 30.261 | 82.615 | 10.343 | 28.0 | 33.75 |
+| 0 | 0 | 2473901162496.0 | 170424302305280.0 | 45.7764 | 30.1483 | 82.923 | 10.382 | 4060086272.0 | 71468255805440.0 |
+| 2500 | 0.0404 | 2192.0 | 21888.0 | 20.8264 | 30.1218 | 82.996 | 10.391 | 1280.0 | 91136.0 |
+| 5000 | 0.0808 | 486.0 | 3120.0 | 18.4124 | 30.4931 | 81.986 | 10.265 | 338.0 | 1128.0 |
+| 7500 | 0.1212 | 276.0 | 1296.0 | 17.0012 | 30.1359 | 82.958 | 10.386 | 255.0 | 250.0 |
+| 10000 | 0.1616 | 202.0 | 756.0 | 16.2078 | 30.1737 | 82.854 | 10.373 | 189.0 | 310.0 |
+| 12500 | 0.2020 | 145.0 | 540.0 | 15.1002 | 30.1534 | 82.909 | 10.38 | 130.0 | 177.0 |
+| 15000 | 0.2424 | 123.5 | 488.0 | 14.5297 | 30.1065 | 83.039 | 10.396 | 93.0 | 149.0 |
+| 17500 | 0.2828 | 95.0 | 376.0 | 14.1518 | 30.1635 | 82.882 | 10.377 | 75.5 | 137.0 |
+| 20000 | 0.3232 | 79.0 | 308.0 | 13.6434 | 30.1294 | 82.976 | 10.389 | 63.0 | 147.0 |
+| 22500 | 0.3636 | 67.0 | 217.0 | 13.1501 | 30.1133 | 83.02 | 10.394 | 51.0 | 83.5 |
+| 25000 | 0.4040 | 63.75 | 203.0 | 12.9684 | 30.2115 | 82.75 | 10.36 | 43.75 | 69.5 |
+| 27500 | 0.4444 | 59.0 | 195.0 | 12.8408 | 30.1817 | 82.832 | 10.371 | 40.0 | 76.0 |
+| 30000 | 0.4848 | 59.0 | 194.0 | 12.8237 | 30.1805 | 82.835 | 10.371 | 41.5 | 55.75 |
+| 32500 | 0.5253 | 58.5 | 174.0 | 12.7748 | 30.1793 | 82.838 | 10.371 | 38.25 | 61.25 |
+| 35000 | 0.5657 | 57.25 | 171.0 | 12.6602 | 30.1387 | 82.95 | 10.385 | 36.25 | 43.75 |
+| 37500 | 0.6061 | 56.25 | 159.0 | 12.5996 | 30.2154 | 82.739 | 10.359 | 36.75 | 49.5 |
+| 40000 | 0.6465 | 55.0 | 156.0 | 12.5526 | 30.1918 | 82.804 | 10.367 | 33.75 | 58.5 |
+| 42500 | 0.6869 | 54.5 | 148.0 | 12.4990 | 30.1749 | 82.85 | 10.373 | 34.0 | 48.25 |
+| 45000 | 0.7273 | 50.75 | 134.0 | 12.2851 | 30.1611 | 82.888 | 10.378 | 29.0 | 42.0 |
+| 47500 | 0.7677 | 51.25 | 126.5 | 12.2410 | 30.2021 | 82.776 | 10.364 | 28.75 | 36.0 |
+| 50000 | 0.8081 | 50.5 | 124.0 | 12.2169 | 30.166 | 82.875 | 10.376 | 28.5 | 38.25 |
+| 52500 | 0.8485 | 49.25 | 122.0 | 12.1910 | 30.1439 | 82.936 | 10.384 | 28.25 | 34.5 |
+| 55000 | 0.8889 | 49.25 | 120.5 | 12.1604 | 30.1149 | 83.015 | 10.394 | 27.625 | 35.0 |
+| 57500 | 0.9293 | 49.25 | 120.5 | 12.1473 | 30.1142 | 83.017 | 10.394 | 27.75 | 33.75 |
+| 60000 | 0.9697 | 49.0 | 119.5 | 12.1403 | 30.143 | 82.938 | 10.384 | 27.625 | 33.5 |
+| 61875 | 1.0 | 49.0 | 120.0 | 12.1391 | 30.2511 | 82.642 | 10.347 | 27.625 | 33.75 |
 
 # Resource Usage Comparison
 
@@ -93,7 +93,7 @@ More information needed
 <br/>
 
 # Train Dataset
-Trained on 145,702,102 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
+Trained on 145,744,973 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
 - Num Samples: `247,500`
 - Subset: `20231101.en`
@@ -103,7 +103,7 @@ Trained on 145,702,102 tokens from the [wikimedia/wikipedia](https://huggingface
 # Training Objective
 
 ```
-DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=5, loss_fn=cos, layer_mapper=layer-2))
+DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=25.0, loss_fn=cos, layer_mapper=layer-2))
 ```
 
 # Hyperparameters
@@ -120,9 +120,9 @@ The following hyperparameters were used during training:
 - lr_scheduler_type: `linear`
 - lr_scheduler_warmup_ratio: `0.5`
 - num_epochs: `1.0`
-- distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=5, loss_fn=cos, layer_mapper=layer-2))`
+- distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=25.0, loss_fn=cos, layer_mapper=layer-2))`
 - train_embeddings: `True`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fbd706f6920>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fbeac4d9c00>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `None`
 - student_model_config: `None`

logs/attn_loss_fn=cos, attn_weight=25.0, layer_mapper=all, projector=orthogonal/events.out.tfevents.1724361746.f383272e719b

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2ad6d9c56daa4463b508abb3dcd40d69b21549e241454d0678caa5f6547b3f3
+size 588