|
--- |
|
language: |
|
- is |
|
- no |
|
- sv |
|
- da |
|
license: cc-by-4.0 |
|
datasets: |
|
- igc |
|
- ic3 |
|
- jonfd/ICC |
|
- mc4 |
|
--- |
|
|
|
# Nordic ELECTRA-Small |
|
This model was pretrained on the following corpora: |
|
* The [Icelandic Gigaword Corpus](http://igc.arnastofnun.is/) (IGC) |
|
* The Icelandic Common Crawl Corpus (IC3) |
|
* The [Icelandic Crawled Corpus](https://huggingface.co/datasets/jonfd/ICC) (ICC) |
|
* The [Multilingual Colossal Clean Crawled Corpus](https://huggingface.co/datasets/mc4) (mC4) - Icelandic, Norwegian, Swedish and Danish text obtained from .is, .no, .se and .dk domains, respectively |
|
|
|
The total size of the corpus after document-level deduplication and filtering was 14.82B tokens, split equally between the four languages. The model was trained using a WordPiece tokenizer with a vocabulary size of 96,105 for one million steps with a batch size of 256, and otherwise with default settings. |
|
|
|
# Acknowledgments |
|
This research was supported with Cloud TPUs from Google's TPU Research Cloud (TRC). |
|
|
|
This project was funded by the Language Technology Programme for Icelandic 2019-2023. The programme, which is managed and coordinated by [Almannarómur](https://almannaromur.is/), is funded by the Icelandic Ministry of Education, Science and Culture. |
