I帽igo L贸pez-Riob贸o Botana
commited on
Commit
路
7ec89f8
1
Parent(s):
523c7a4
Update README.md
Browse files
README.md
CHANGED
@@ -106,7 +106,7 @@ You can check the [original GitHub repository](https://github.com/microsoft/Dial
|
|
106 |
- This model uses the original English-based tokenizer from [GPT-2 paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf).
|
107 |
Spanish tokenization is not considered but it has similarities in grammatical structure for encoding text. This overlap may help the model transfer its knowledge from English to Spanish.
|
108 |
Moreover, the BPE (Byte Pair Encoding) implementation of the GPT-2 tokenizer **can assign a representation to every Unicode string**.
|
109 |
-
**From the GPT-2 paper**:
|
110 |
> Since our approach can assign a probability to any Unicode string, this allows us to evaluate our LMs on any dataset regardless of pre-processing, tokenization, or vocab size.
|
111 |
- This model is intended to be used **just for single-turn chitchat conversations in Spanish**.
|
112 |
- This model's generation capabilities are limited to the extent of the aforementioned fine-tuning dataset.
|
|
|
106 |
- This model uses the original English-based tokenizer from [GPT-2 paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf).
|
107 |
Spanish tokenization is not considered but it has similarities in grammatical structure for encoding text. This overlap may help the model transfer its knowledge from English to Spanish.
|
108 |
Moreover, the BPE (Byte Pair Encoding) implementation of the GPT-2 tokenizer **can assign a representation to every Unicode string**.
|
109 |
+
**From the GPT-2 paper**:
|
110 |
> Since our approach can assign a probability to any Unicode string, this allows us to evaluate our LMs on any dataset regardless of pre-processing, tokenization, or vocab size.
|
111 |
- This model is intended to be used **just for single-turn chitchat conversations in Spanish**.
|
112 |
- This model's generation capabilities are limited to the extent of the aforementioned fine-tuning dataset.
|