---
language: ["ru", "en"]
tags:
- russian
- fill-mask
- pretraining
- embeddings
- masked-lm
- tiny
license: mit
widget:
- text: "Миниатюрная модель для [MASK] разных задач."
---
This is a very small distilled version of the [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased) model for Russian and English.

This model is useful if you want to fine-tune it for a relatively simple Russian task (e.g. NER or sentiment classification), and you care more about speed and size than about accuracy. It is approximately x10 smaller and faster than [DeepPavlov/rubert-base-cased-sentence](https://huggingface.co/DeepPavlov/rubert-base-cased-sentence).

It was trained on the [Yandex Translate corpus](https://translate.yandex.ru/corpus) using MLM loss (partially distilled from [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased)) and translation ranking loss (partially distilled from [LaBSE](https://huggingface.co/sentence-transformers/LaBSE)).