jordimas commited on
Commit
257f741
·
1 Parent(s): c7e3def

Initial version

Browse files
Files changed (6) hide show
  1. README.md +58 -3
  2. config.json +2 -0
  3. model.bin +3 -0
  4. model_description.txt +9 -0
  5. shared_vocabulary.txt +0 -0
  6. sp_m.model +3 -0
README.md CHANGED
@@ -1,3 +1,58 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - ca
4
+ - gl
5
+
6
+ tags:
7
+ - translation
8
+
9
+ library_name: opennmt
10
+ license: mit
11
+ metrics:
12
+ - bleu
13
+
14
+ inference: false
15
+ ---
16
+
17
+ ### Introduction
18
+
19
+ Galician - Catalan translation model based on OpenNMT. These are the same models that we have in production at https://www.softcatala.org/traductor/.
20
+
21
+
22
+ ### Usage
23
+
24
+
25
+ ```bash
26
+ pip3 install ctranslate2 pyonmttok
27
+ ```
28
+
29
+ Simple translation using Python:
30
+
31
+
32
+
33
+ ```python
34
+ import ctranslate2
35
+ import pyonmttok
36
+ from huggingface_hub import snapshot_download
37
+ model_dir = snapshot_download(repo_id="softcatala/translate-glg-cat", revision="main")
38
+
39
+ tokenizer=pyonmttok.Tokenizer(mode="none", sp_model_path = model_dir + "/sp_m.model")
40
+ tokenized=tokenizer.tokenize("Bom día a todos")
41
+
42
+ import ctranslate2
43
+ translator = ctranslate2.Translator(model_dir)
44
+ translated = translator.translate_batch([tokenized[0]])
45
+ print(tokenizer.detokenize(translated[0][0]['tokens']))
46
+ ```
47
+
48
+ ## Benchmarks
49
+
50
+ | testset | BLEU |
51
+ |---------------------------------------|-------|
52
+ | test dataset (from train/dev/test) | 74.1 |
53
+ | Flores200 dataset | 31.4 |
54
+
55
+ ## Additional information
56
+ * https://github.com/Softcatala/nmt-models
57
+ * https://github.com/Softcatala/parallel-catalan-corpus
58
+
config.json ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ {
2
+ }
model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d88b49a20ced6e0581b08714285f097d132437be725ae52a9ccdc462265484b3
3
+ size 70727761
model_description.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ Model description: glg-cat
2
+ Date: 2022-11-17
3
+ TF version 2.10.0, OpenNMT version 2.29.1, CTranslate2 version 2.24.0
4
+ Test data set
5
+ BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0 = 74.1 88.8/79.3/72.2/66.0 (BP = 0.973 ratio = 0.974 hyp_len = 102921 ref_len = 105694)
6
+ chrF2|nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0 = 86.2
7
+ Flores data set
8
+ BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.1.0 = 31.4 61.4/37.7/24.9/16.8 (BP = 1.000 ratio = 1.026 hyp_len = 28017 ref_len = 27304)
9
+ chrF2|nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.1.0 = 59.9
shared_vocabulary.txt ADDED
The diff for this file is too large to render. See raw diff
 
sp_m.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ed20468df719c191291ee60835e235af1bc7a67a6a270ae936bd4ec868f81c98
3
+ size 1136059