Update README.md
Browse files
README.md
CHANGED
|
@@ -7,6 +7,8 @@ base_model: alpindale/WizardLM-2-8x22B
|
|
| 7 |
|
| 8 |
Oh boy, here we go. Low-rank (`r=16, alpha=32`) LoRA on top of [WizardLM-2-8x22B](https://huggingface.co/alpindale/WizardLM-2-8x22B), trained on 2 epochs of (cleaned & deduped) c2-logs. As far as I can tell, this is an upgrade from `WizardLM-2-8x22B` for RP purposes.
|
| 9 |
|
|
|
|
|
|
|
| 10 |
## Why A LoRA?
|
| 11 |
|
| 12 |
The choice was fully intentional. I briefly considered a FFT but for this particular use-case a LoRA seemed a better fit. `WizardLM-2-8x22B` is smart by itself but its used vocabulary leaves much to be desired when it comes to RP. By training a low-rank LoRA on top of it to teach it some of Claude's writing style, we remedy that.
|
|
|
|
| 7 |
|
| 8 |
Oh boy, here we go. Low-rank (`r=16, alpha=32`) LoRA on top of [WizardLM-2-8x22B](https://huggingface.co/alpindale/WizardLM-2-8x22B), trained on 2 epochs of (cleaned & deduped) c2-logs. As far as I can tell, this is an upgrade from `WizardLM-2-8x22B` for RP purposes.
|
| 9 |
|
| 10 |
+
Alongside this ready-to-use release there is also two alternatives released that are "mergebait", `rAIfle/sorcLM-epoch1-bf16` and `rAIfle/sorcLM-epoch2-bf16`. These are very experimental and not fit for use, but could be interesting for merging. Refer to their respective readmes for more info.
|
| 11 |
+
|
| 12 |
## Why A LoRA?
|
| 13 |
|
| 14 |
The choice was fully intentional. I briefly considered a FFT but for this particular use-case a LoRA seemed a better fit. `WizardLM-2-8x22B` is smart by itself but its used vocabulary leaves much to be desired when it comes to RP. By training a low-rank LoRA on top of it to teach it some of Claude's writing style, we remedy that.
|