README.md

---
language: de
license: mit
datasets: cc100
---

# xlm-roberta-base-focus-german

XLM-R adapted to German using "FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models".

Code: https://github.com/konstantinjdobler/focus

Paper: https://arxiv.org/abs/2305.14481

## Usage
```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("konstantindobler/xlm-roberta-base-focus-german")
model = AutoModelForMaskedLM.from_pretrained("konstantindobler/xlm-roberta-base-focus-german")

# Use model and tokenizer as usual
```

## Details
The model is based on [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) and was adapted to German.
The original multilingual tokenizer was replaced by a language-specific German tokenizer with a vocabulary of 50k tokens. The new embeddings were initialized with FOCUS.
The model was then trained on data from CC100 for 390k optimizer steps. More details and hyperparameters can be found [in the paper](https://arxiv.org/abs/2305.14481).

## Disclaimer
The web-scale dataset used for pretraining and tokenizer training (CC100) might contain personal and sensitive information.
Such behavior needs to be assessed carefully before any real-world deployment of the models. Also, the tokenizer training was conducted using a sentencepiece `character_coverage` of 100%. As a result, the vocabulary contains characters which are usually not used in German.

## Citation
Please cite FOCUS as follows:

```bibtex
@misc{dobler-demelo-2023-focus,
    title={FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models},
    author={Konstantin Dobler and Gerard de Melo},
    year={2023},
    eprint={2305.14481},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```