End of training

README.md

@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 685.3206
-- eval_frwikippl: 4166.5459
-- eval_zhwikippl: 10016.8096
-- eval_loss: 7038.3359
-- eval_runtime: 21.4586
-- eval_samples_per_second: 46.601
-- eval_steps_per_second: 11.65
+- eval_enwikippl: 611.0229
+- eval_frwikippl: 4057.1116
+- eval_zhwikippl: 17857.3535
+- eval_loss: 7319.6802
+- eval_runtime: 21.6635
+- eval_samples_per_second: 46.161
+- eval_steps_per_second: 11.54
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -45,10 +45,10 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- distillation_objective: <distily.objectives.LegacyObjective object at 0x7f7d9010c220>
+- distillation_objective: <distily.objectives.LegacyObjective object at 0x7f7f68372f20>
 - train_embeddings: True
 - learning_rate: 4e-05
-- train_batch_size: 8
+- train_batch_size: 4
 - eval_batch_size: 4
 - seed: 42
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
@@ -62,32 +62,57 @@ Peak GPU Memory: 15.7299 GB
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 30.2385 | 57.2728 |  |  |  |  | 18.1772 |
-| 0 | 0 | 55339.3672 | 57682.5742 | 331776.0 | 21.5939 | 46.309 | 11.577 | 57080.2930 |
-| 500 | 0.0404 | 2539.2073 | 10701.1621 | 12591.8076 | 21.7416 | 45.995 | 11.499 | 52355.7031 |
-| 1000 | 0.0808 | 1909.6178 | 6915.1094 | 10639.6162 | 21.6609 | 46.166 | 11.542 | 28701.2539 |
-| 1500 | 0.1212 | 1563.3774 | 6141.7974 | 9791.8721 | 21.4972 | 46.518 | 11.629 | 22309.7324 |
-| 2000 | 0.1616 | 1355.0397 | 6279.7275 | 9359.6162 | 21.5918 | 46.314 | 11.578 | 21985.8770 |
-| 2500 | 0.2020 | 1227.5216 | 5683.0625 | 9057.0879 | 21.5112 | 46.487 | 11.622 | 18872.3887 |
-| 3000 | 0.2424 | 1126.7594 | 5939.9272 | 8646.5283 | 21.3909 | 46.749 | 11.687 | 24154.6621 |
-| 3500 | 0.2828 | 1050.4360 | 5464.5869 | 8420.1602 | 21.4858 | 46.542 | 11.636 | 17743.2598 |
-| 4000 | 0.3232 | 995.7695 | 5204.1860 | 8302.2080 | 21.671 | 46.145 | 11.536 | 16421.9824 |
-| 4500 | 0.3636 | 939.8518 | 4812.9336 | 8087.6162 | 21.6884 | 46.108 | 11.527 | 18503.0742 |
-| 5000 | 0.4040 | 894.0764 | 5064.1064 | 7913.1841 | 21.5518 | 46.4 | 11.6 | 18140.9707 |
-| 5500 | 0.4444 | 854.5688 | 4709.2144 | 7854.3359 | 21.4663 | 46.585 | 11.646 | 13195.6348 |
-| 6000 | 0.4848 | 815.7767 | 4654.2524 | 7642.8481 | 21.5227 | 46.463 | 11.616 | 14954.4814 |
-| 6500 | 0.5253 | 795.4309 | 4827.8882 | 7615.8398 | 21.7578 | 45.961 | 11.49 | 16576.2129 |
-| 7000 | 0.5657 | 769.8770 | 4643.7627 | 7491.7759 | 21.5078 | 46.495 | 11.624 | 16412.1191 |
-| 7500 | 0.6061 | 755.6475 | 4527.0581 | 7376.8638 | 21.6221 | 46.249 | 11.562 | 15845.7139 |
-| 8000 | 0.6465 | 728.3958 | 4527.8569 | 7300.3838 | 21.828 | 45.813 | 11.453 | 17439.0645 |
-| 8500 | 0.6869 | 725.3337 | 4182.8813 | 7177.5039 | 21.5139 | 46.482 | 11.62 | 14905.6289 |
-| 9000 | 0.7273 | 693.5788 | 4218.2700 | 7109.6958 | 21.5247 | 46.458 | 11.615 | 13303.5693 |
-| 9500 | 0.7677 | 685.3206 | 4166.5459 | 7038.3359 | 21.4586 | 46.601 | 11.65 | 10016.8096 |
-| 10000 | 0.8081 | 681.6716 | 4036.2834 | 7004.8638 | 21.5314 | 46.444 | 11.611 | 8738.9414 |
-| 10500 | 0.8485 | 656.0428 | 4098.0786 | 6922.6558 | 21.6176 | 46.259 | 11.565 | 9690.5127 |
-| 11000 | 0.8889 | 655.4447 | 4287.2363 | 6842.6240 | 21.4458 | 46.629 | 11.657 | 13403.4229 |
-| 11500 | 0.9293 | 635.5220 | 4204.7627 | 6807.0400 | 21.4021 | 46.724 | 11.681 | 10938.9814 |
-| 12000 | 0.9697 | 633.3297 | 4234.2144 | 6815.6162 | 21.5151 | 46.479 | 11.62 | 10046.2852 |
-| 12375 | 1.0 | 633.1452 | 4051.2502 | 6749.3442 | 21.4393 | 46.643 | 11.661 | 10211.2959 |
+| 0 | 0 | 54063.0 | 57132.0859 | 330301.4375 | 21.4102 | 46.707 | 11.677 | 54288.6797 |
+| 500 | 0.0202 | 2200.8281 | 11269.5508 | 12821.6318 | 21.5139 | 46.482 | 11.62 | 56764.8438 |
+| 1000 | 0.0404 | 1665.7762 | 6195.2891 | 10439.8076 | 21.4414 | 46.639 | 11.66 | 26684.8594 |
+| 1500 | 0.0606 | 1395.5389 | 6104.2388 | 9813.1201 | 21.6061 | 46.283 | 11.571 | 24228.9746 |
+| 2000 | 0.0808 | 1240.8671 | 5769.0522 | 9419.9043 | 21.4266 | 46.671 | 11.668 | 25033.0820 |
+| 2500 | 0.1010 | 1090.4764 | 4890.7593 | 9141.3115 | 21.621 | 46.251 | 11.563 | 20563.0996 |
+| 3000 | 0.1212 | 1002.9867 | 4993.7222 | 8898.1758 | 21.5727 | 46.355 | 11.589 | 19028.0254 |
+| 3500 | 0.1414 | 938.0837 | 5002.0063 | 8604.9277 | 21.6847 | 46.115 | 11.529 | 16472.4883 |
+| 4000 | 0.1616 | 887.3153 | 5051.4448 | 8486.5283 | 21.6081 | 46.279 | 11.57 | 16085.5645 |
+| 4500 | 0.1818 | 845.5585 | 4589.0835 | 8366.6562 | 21.4849 | 46.544 | 11.636 | 15081.8213 |
+| 5000 | 0.2020 | 793.8573 | 4711.3706 | 8115.8398 | 21.5826 | 46.334 | 11.583 | 17669.9570 |
+| 5500 | 0.2222 | 769.3093 | 4488.2871 | 8050.4321 | 21.6441 | 46.202 | 11.55 | 14243.0586 |
+| 6000 | 0.2424 | 728.3535 | 4591.8325 | 7877.5039 | 21.5753 | 46.349 | 11.587 | 17997.4102 |
+| 6500 | 0.2626 | 710.1467 | 4239.1436 | 7786.5601 | 21.6028 | 46.29 | 11.573 | 13550.0938 |
+| 7000 | 0.2828 | 687.4926 | 4355.0205 | 7698.4961 | 21.646 | 46.198 | 11.549 | 17154.9648 |
+| 7500 | 0.3030 | 671.2838 | 4329.4575 | 7575.6479 | 21.6404 | 46.21 | 11.552 | 14209.8184 |
+| 8000 | 0.3232 | 651.2464 | 4281.1934 | 7469.9521 | 21.5348 | 46.437 | 11.609 | 11029.1934 |
+| 8500 | 0.3434 | 644.4048 | 4005.5222 | 7476.8638 | 21.677 | 46.132 | 11.533 | 14790.6338 |
+| 9000 | 0.3636 | 617.8220 | 3900.7300 | 7369.7598 | 21.5714 | 46.358 | 11.589 | 14878.7871 |
+| 9500 | 0.3838 | 611.0229 | 4057.1116 | 7319.6802 | 21.6635 | 46.161 | 11.54 | 17857.3535 |
+| 10000 | 0.4040 | 607.8168 | 4144.8633 | 7314.2402 | 21.6408 | 46.209 | 11.552 | 8414.2871 |
+| 10500 | 0.4242 | 597.2198 | 3698.7766 | 7203.5840 | 21.435 | 46.653 | 11.663 | 6787.8618 |
+| 11000 | 0.4444 | 583.0488 | 3682.1257 | 7166.8481 | 21.6697 | 46.147 | 11.537 | 8091.4233 |
+| 11500 | 0.4646 | 571.3620 | 3917.5435 | 7146.4961 | 21.6445 | 46.201 | 11.55 | 6319.4146 |
+| 12000 | 0.4848 | 573.1504 | 3768.2695 | 7058.4961 | 21.5215 | 46.465 | 11.616 | 7570.3271 |
+| 12500 | 0.5051 | 566.9861 | 3949.1577 | 7061.6318 | 22.0963 | 45.256 | 11.314 | 8727.2812 |
+| 13000 | 0.5253 | 558.8017 | 3803.5042 | 6961.4082 | 21.5517 | 46.4 | 11.6 | 9476.1670 |
+| 13500 | 0.5455 | 550.8849 | 3855.4763 | 7017.2798 | 21.6135 | 46.267 | 11.567 | 11842.0234 |
+| 14000 | 0.5657 | 546.9738 | 3748.3950 | 6947.0718 | 21.5141 | 46.481 | 11.62 | 11424.9463 |
+| 14500 | 0.5859 | 535.9684 | 3708.3093 | 6870.6240 | 21.6892 | 46.106 | 11.526 | 9886.5801 |
+| 15000 | 0.6061 | 528.7446 | 3590.0920 | 6851.8398 | 21.6211 | 46.251 | 11.563 | 14917.5742 |
+| 15500 | 0.6263 | 521.6382 | 3602.8992 | 6849.3442 | 21.4696 | 46.577 | 11.644 | 10334.7578 |
+| 16000 | 0.6465 | 517.2215 | 3595.4131 | 6779.7759 | 21.4612 | 46.596 | 11.649 | 13237.1143 |
+| 16500 | 0.6667 | 518.6794 | 3385.7922 | 6766.4639 | 21.7793 | 45.915 | 11.479 | 10972.6348 |
+| 17000 | 0.6869 | 515.1771 | 3393.7964 | 6732.6719 | 21.9049 | 45.652 | 11.413 | 9149.7510 |
+| 17500 | 0.7071 | 501.9872 | 3414.7986 | 6733.2480 | 21.6388 | 46.213 | 11.553 | 6785.5962 |
+| 18000 | 0.7273 | 501.7729 | 3411.7908 | 6693.3442 | 21.6999 | 46.083 | 11.521 | 5611.2793 |
+| 18500 | 0.7475 | 497.0320 | 3345.2139 | 6665.5361 | 21.7517 | 45.973 | 11.493 | 6820.1172 |
+| 19000 | 0.7677 | 494.9137 | 3287.4609 | 6658.4321 | 21.5993 | 46.298 | 11.574 | 8307.6650 |
+| 19500 | 0.7879 | 489.8856 | 3310.1404 | 6703.3921 | 21.703 | 46.077 | 11.519 | 7612.3970 |
+| 20000 | 0.8081 | 485.6058 | 3133.0156 | 6542.2720 | 21.6927 | 46.098 | 11.525 | 6796.0293 |
+| 20500 | 0.8283 | 482.7485 | 3159.1875 | 6622.9761 | 21.8265 | 45.816 | 11.454 | 6659.4741 |
+| 21000 | 0.8485 | 469.4856 | 3162.3074 | 6523.4561 | 21.7088 | 46.064 | 11.516 | 11412.7490 |
+| 21500 | 0.8687 | 475.0967 | 3214.5735 | 6542.4639 | 21.8362 | 45.796 | 11.449 | 8429.4746 |
+| 22000 | 0.8889 | 471.5591 | 3336.0288 | 6522.2080 | 21.7328 | 46.013 | 11.503 | 5576.1724 |
+| 22500 | 0.9091 | 467.7843 | 3244.9744 | 6460.9600 | 21.7546 | 45.967 | 11.492 | 3847.1572 |
+| 23000 | 0.9293 | 459.4413 | 3435.3245 | 6459.0400 | 21.7279 | 46.024 | 11.506 | 5404.9829 |
+| 23500 | 0.9495 | 466.1707 | 3223.0857 | 6445.9839 | 21.8014 | 45.869 | 11.467 | 6019.9951 |
+| 24000 | 0.9697 | 451.2469 | 3094.4868 | 6465.6318 | 21.8221 | 45.825 | 11.456 | 5374.3945 |
+| 24500 | 0.9899 | 461.2016 | 3168.5593 | 6462.3042 | 21.5588 | 46.385 | 11.596 | 7845.6411 |
+| 24750 | 1.0 | 453.6186 | 3034.7439 | 6427.3281 | 21.8641 | 45.737 | 11.434 | 7226.5781 |
 
 ### Framework versions
 - Distily 0.2.0

logs/per_device_train_batch_size=4/events.out.tfevents.1723370138.93d6cbb3ad53

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eae42ba4b31f101de5dfd727dea08ce460ab7bcd606f625aa06360d79f5f640c
+size 253