Update README.md
Browse files
README.md
CHANGED
|
@@ -14,30 +14,6 @@ tags:
|
|
| 14 |
---
|
| 15 |
for testing purposes only. qlora trained using peft on codellama/CodeLlama-7b-hf as base model. trained on mrm8488/unnatural-instructions, config 'core' dataset.
|
| 16 |
|
| 17 |
-
trained at 1000 steps with checkpoint every 50. training/validation loss below:
|
| 18 |
-
|
| 19 |
-
Step Training Loss Validation Loss
|
| 20 |
-
50 1.480500 0.935647
|
| 21 |
-
100 0.894800 0.867328
|
| 22 |
-
150 0.835700 0.841386
|
| 23 |
-
200 0.846100 0.823671
|
| 24 |
-
250 0.804600 0.791546
|
| 25 |
-
300 0.744000 0.799941
|
| 26 |
-
350 0.721900 0.707534
|
| 27 |
-
400 0.702700 0.697420
|
| 28 |
-
450 0.698200 0.691702
|
| 29 |
-
500 0.674600 0.687037
|
| 30 |
-
550 0.666700 0.683634
|
| 31 |
-
600 0.687200 0.680872
|
| 32 |
-
650 0.679300 0.677384
|
| 33 |
-
700 0.698900 0.675221
|
| 34 |
-
750 0.652500 0.673152
|
| 35 |
-
800 0.672200 0.671620
|
| 36 |
-
850 0.668700 0.669980
|
| 37 |
-
900 0.638100 0.669189
|
| 38 |
-
950 0.663200 0.668443
|
| 39 |
-
1000 0.668300 0.668069
|
| 40 |
-
|
| 41 |
training data transformed to the following structure for testing purposes:
|
| 42 |
```Example 1:
|
| 43 |
Input: <s>[INST] <<SYS>>
|
|
|
|
| 14 |
---
|
| 15 |
for testing purposes only. qlora trained using peft on codellama/CodeLlama-7b-hf as base model. trained on mrm8488/unnatural-instructions, config 'core' dataset.
|
| 16 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
training data transformed to the following structure for testing purposes:
|
| 18 |
```Example 1:
|
| 19 |
Input: <s>[INST] <<SYS>>
|