Hugging Face's logo

BramVanroy
/
kenlm_wikipedia_nl

Dutch
kenlm
Model card Files Community
kenlm_wikipedia_nl
File size: 2,525 Bytes 
ae4a669
 
 
 
 
 
 
0b9a2ce
 
ae4a669
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
 
---
language:
- nl
tags:
- kenlm
license: apache-2.0
---


# KenLM (arpa) models for Dutch based on Wikipedia

This repository contains KenLM models (n=5) for Dutch, based on the [Dutch portion of Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia/viewer/20231101.nl) - sentence-segmented (one sentence per line). Models are provided on tokens, part-of-speech, dependency labels, and lemmas, as processed with spaCy `nl_core_news_sm`:

- wiki_nl_token.arpa[.bin]: token
- wiki_nl_pos.arpa[.bin]: part-of-speech tag
- wiki_nl_dep.arpa[.bin]: dependency label
- wiki_nl_lemma.arpa[.bin]: lemma

Both regular `.arpa` files as well as more efficient KenLM binary files (`.arpa.bin`) are provided. You probably want to use the binary versions.

## Usage from within Python

Make sure to install dependencies:

```shell
pip install huggingface_hub
pip install https://github.com/kpu/kenlm/archive/master.zip

# If you want to use spaCy preprocessing
pip install spacy
python -m spacy download nl_core_news_sm
```

We can then use the Hugging Face hub software to download and cache the model file that we want, and directly use it with KenLM.

```python
import kenlm
from huggingface_hub import hf_hub_download

model_file = hf_hub_download(repo_id="BramVanroy/kenlm_wikipedia_nl", filename="wiki_nl_token.arpa.bin")
model = kenlm.Model(model_file)

text = "Ik eet graag koekjes !"  # pre-tokenized
model.perplexity(text)
# 1790.5033832700467
```

It is recommended to use spaCy as a preprocessor to automatically use the same tagsets and tokenization as were used when creating the LMs.


```python
import kenlm
import spacy
from huggingface_hub import hf_hub_download

model_file = hf_hub_download(repo_id="BramVanroy/kenlm_wikipedia_nl", filename="wiki_nl_pos.arpa.bin")  # pos file
model = kenlm.Model(model_file)

nlp = spacy.load("nl_core_news_sm")

text = "Ik eet graag koekjes!" 
pos_sequence = " ".join([token.pos_ for token in nlp(text)])
# 'PRON VERB ADV NOUN PUNCT'
model.perplexity(pos_sequence)
# 6.190638021041525
```


## Reproduction

Example: 

```sh
bin/lmplz -o 5 -S 75% -T ../data/tmp/ < ../data/wikipedia/nl/wiki_nl_processed_lemma_dedup.txt > ../data/wikipedia/nl/models/wiki_nl_lemma.arpa
bin/build_binary ../data/wikipedia/nl/models/wiki_nl_lemma.arpa ../data/wikipedia/nl/models/wiki_nl_lemma.arpa.bin
```

For class-based LMs (POS and DEP), the `--discount_fallback` was used and the parsed data was not deduplicated (but it was deduplicated on the sentence-level for token and lemma models).