DaryaTereshchenko commited on
Commit
2d9442e
·
1 Parent(s): c11de4c

Updated README file with new information

Browse files
Files changed (1) hide show
  1. README.md +65 -1
README.md CHANGED
@@ -40,4 +40,68 @@ language:
40
  - uk
41
  - vi
42
  - zh
43
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
  - uk
41
  - vi
42
  - zh
43
+ ---
44
+
45
+ # PRISM Model for Multilingual Machine Translation
46
+
47
+ This repository contains the `Prism` model, a state-of-the-art multilingual neural machine translation (NMT) system developed for translation. The `Prism` model supports translation across 39 languages, leveraging a zero-shot paraphrasing approach that does not require human judgments for training.
48
+
49
+ The model was trained with a focus on multilingual performance, excelling in tasks such as translation quality estimation and evaluation, making it a versatile choice for research and practical use in various language pairs.
50
+
51
+ ## Model Description
52
+
53
+ The `Prism` model was designed to be a lexically/syntactically unbiased paraphraser. The core idea is to treat paraphrasing as a zero-shot translation task, which allows the model to cover a wide range of languages effectively.
54
+
55
+ ### BLEU Score Performance
56
+
57
+ Based on the research paper, the `Prism` model achieved competitive or superior performance across various language pairs in the WMT 2019 shared metrics task. It outperformed existing evaluation metrics in many cases, showing robustness in both high-resource and low-resource settings.
58
+
59
+ ## Installation
60
+
61
+ To use `PrismTokenizer`, ensure that the `sentencepiece` package is installed, as it is a required dependency for handling multilingual tokenization.
62
+
63
+ ```bash
64
+ pip install sentencepiece
65
+ ```
66
+
67
+ ## Usage Example
68
+ ```python
69
+ from transformers import PrismForConditionalGeneration, PrismTokenizer
70
+
71
+ uk_text = "Життя як коробка шоколаду"
72
+ ja_text = "人生はチョコレートの箱のようなもの。"
73
+
74
+ model = PrismForConditionalGeneration.from_pretrained("facebook/prism")
75
+ tokenizer = PrismTokenizer.from_pretrained("facebook/prism")
76
+
77
+ # Translate Ukrainian to French
78
+ tokenizer.src_lang = "uk"
79
+ encoded_uk = tokenizer(uk_text, return_tensors="pt")
80
+ generated_tokens = model.generate(**encoded_uk, forced_bos_token_id=tokenizer.get_lang_id("fr"), max_new_tokens=20)
81
+ print(tokenizer.batch_decode(generated_tokens, skip_special_tokens=True))
82
+ # => '<fr> La vie comme une boîte de chocolat.'
83
+
84
+ # Translate Japanese to English
85
+ tokenizer.src_lang = "ja"
86
+ encoded_ja = tokenizer(ja_text, return_tensors="pt")
87
+ generated_tokens = model.generate(**encoded_ja, forced_bos_token_id=tokenizer.get_lang_id("en"), max_new_tokens=20)
88
+ print(tokenizer.batch_decode(generated_tokens, skip_special_tokens=True))
89
+ # => '<en> Life is like a box of chocolate.'
90
+ ```
91
+
92
+ ## Languages Covered
93
+ Albanian (sq), Arabic (ar), Bengali (bn), Bulgarian (bg), Catalan; Valencian (ca), Chinese (zh), Croatian (hr), Czech (cs), Danish (da), Dutch (nl), English (en), Esperanto (eo), Estonian (et), Finnish (fi), French (fr), German (de), Greek, Modern (el), Hebrew (modern) (he), Hungarian (hu), Indonesian (id), Italian (it), Japanese (ja), Kazakh (kk), Latvian (lv), Lithuanian (lt), Macedonian (mk), Norwegian (no), Polish (pl), Portuguese (pt), Romanian, Moldovan (ro), Russian (ru), Serbian (sr), Slovak (sk), Slovene (sl), Spanish; Castilian (es), Swedish (sv), Turkish (tr), Ukrainian (uk), Vietnamese (vi).
94
+
95
+ ## Citation
96
+ If you use this model in your research, please cite the original paper:
97
+ ```
98
+ @inproceedings{thompson-post-2020-automatic,
99
+ title={Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing},
100
+ author={Brian Thompson and Matt Post},
101
+ year={2020},
102
+ booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
103
+ month = nov,
104
+ address = "Online",
105
+ publisher = "Association for Computational Linguistics",
106
+ }
107
+ ```