|
--- |
|
license: other |
|
license_name: microsoft |
|
license_link: https://huggingface.co/microsoft/Orca-2-13b/blob/main/LICENSE |
|
--- |
|
|
|
Lost of good models use Orca for their merges, however vanilla Orca has vocabulary size of 32003, |
|
3 last tokens are ChatML tokens and a PAD token. This causes errors during a merge with models with standard 32000 vocabulary size. |
|
|
|
I've removed those tokens from volabulary and resized model embeddings to mach 32000 standard size. So this model is ready to be used |
|
as a merge component in mergekit. It may not work on its own with ChatML template anymore. |
|
|
|
```python |
|
model.resize_token_embeddings(32000) |
|
``` |
