metadata
language:
- ru
- en
tags:
- russian
- fill-mask
- pretraining
- embeddings
- masked-lm
- tiny
license: mit
widget:
- text: Миниатюрная модель для [MASK] разных задач.
This is a very small distilled version of the bert-base-multilingual-cased model for Russian and English.
This model is useful if you want to fine-tune it for a relatively simple Russian task (e.g. NER or sentiment classification), and you care more about speed and size than about accuracy. It is approximately x10 smaller and faster than DeepPavlov/rubert-base-cased-sentence.
It was trained on the Yandex Translate corpus using MLM loss (partially distilled from bert-base-multilingual-cased) and translation ranking loss (partially distilled from LaBSE).