Update README.md
Browse files
README.md
CHANGED
|
@@ -10,7 +10,7 @@ tags:
|
|
| 10 |
This is not an instruct fine tune, instead it's an attempt to de-contaminate the model, remove gptslop and refusals. I want model to feel like it was trained on human data, not synthetic one.
|
| 11 |
|
| 12 |
|
| 13 |
-
About 961 steps total, Yi-34B-200K llamafied DPO trained for 1 epoch on rawrr_v2 dataset via unsloth at prompt length of 400 and max length of 700, lr 0.000045 \
|
| 14 |
Model initialized with max_positional_embeddings of 4096 to not OOM. \
|
| 15 |
Training done on RTX 3090 Ti in about 14 hours. \
|
| 16 |
Average mem usage was like 23.89 / 23.99 GiB, so very close to OOM at all times. \
|
|
|
|
| 10 |
This is not an instruct fine tune, instead it's an attempt to de-contaminate the model, remove gptslop and refusals. I want model to feel like it was trained on human data, not synthetic one.
|
| 11 |
|
| 12 |
|
| 13 |
+
About 961 steps total, Yi-34B-200K llamafied DPO trained for 1 epoch on rawrr_v2 dataset via unsloth qlora at prompt length of 400 and max length of 700, lr 0.000045 \
|
| 14 |
Model initialized with max_positional_embeddings of 4096 to not OOM. \
|
| 15 |
Training done on RTX 3090 Ti in about 14 hours. \
|
| 16 |
Average mem usage was like 23.89 / 23.99 GiB, so very close to OOM at all times. \
|