PM-AI
/

sts_paraphrase_xlm-roberta-base_de-en

@@ -1,94 +1,141 @@
 ---
 pipeline_tag: sentence-similarity
 tags:
-- sentence-transformers
 - feature-extraction
-- sentence-similarity
 - transformers
 ---
-# PM-AI/sts_paraphrase_xlm-roberta-base_de-en
-This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
-<!--- Describe your model here -->
-## Usage (Sentence-Transformers)
-Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
-```
-pip install -U sentence-transformers
-```
-Then you can use the model like this:
-```python
-from sentence_transformers import SentenceTransformer
-sentences = ["This is an example sentence", "Each sentence is converted"]
-model = SentenceTransformer('PM-AI/sts_paraphrase_xlm-roberta-base_de-en')
-embeddings = model.encode(sentences)
-print(embeddings)
-```
-## Usage (HuggingFace Transformers)
-Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
-```python
-from transformers import AutoTokenizer, AutoModel
-import torch
-#Mean Pooling - Take attention mask into account for correct averaging
-def mean_pooling(model_output, attention_mask):
-    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
-    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
-    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
-# Sentences we want sentence embeddings for
-sentences = ['This is an example sentence', 'Each sentence is converted']
-# Load model from HuggingFace Hub
-tokenizer = AutoTokenizer.from_pretrained('PM-AI/sts_paraphrase_xlm-roberta-base_de-en')
-model = AutoModel.from_pretrained('PM-AI/sts_paraphrase_xlm-roberta-base_de-en')
-# Tokenize sentences
-encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
-# Compute token embeddings
-with torch.no_grad():
-    model_output = model(**encoded_input)
-# Perform pooling. In this case, mean pooling.
-sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
-print("Sentence embeddings:")
-print(sentence_embeddings)
-```
-## Evaluation Results
-<!--- Describe how your model was evaluated -->
-For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=PM-AI/sts_paraphrase_xlm-roberta-base_de-en)
-## Full Model Architecture
-```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel
-  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
-)
-```
-## Citing & Authors
-<!--- Describe where people can find more information -->

 ---
+language:
+- de
+- en
+datasets:
+- todo
 pipeline_tag: sentence-similarity
 tags:
+- semantic textual similarity
+- sts
+- semantic search
+- sentence similarity
+- paraphrasing
+- sentence-transformer
 - feature-extraction
 - transformers
+task_categories:
+- sentence-similarity
+- feature-extraction
+- text-retrieval
+- other
 ---
+# Model card for PM-AI/sts_paraphrase_xlm-roberta-base_de-en
+## Model summary
+Transformer model for **Semantic Textual Similarity (STS)** for _German_ and _Englisch_ sentences/texts.
+The embeddings output can be used for **semantic search**, **paraphrasing** and **retrieval** with _cosine similarity_.
+The Model is applicable to _Englisch-German-Mixed_ sentences/texts but also for _Englisch only_ and _German only_.
+The model can be easily used with [Sentence Transformer](https://github.com/UKPLab/sentence-transformers) library.
+## Training
+This model is based on a training approach from 2020 by Philip May, who published the [T-Systems-onsite/cross-en-de-roberta-sentence-transformer](https://huggingface.co/T-Systems-onsite/cross-en-de-roberta-sentence-transformer) model.
+We updated this approach by a new base model and some extensions to the training data.
+These changes are discussed in the next sections.
+### Training Data
+The model is based on training with samples from STSb, SICK and Priya22 semantic textual relatedness datasets.
+It contains about 76.000 sentence pairs.
+These sentence pairs are based on German-German, English-English and German-English mixed.
+The training object is to optimize for cosine similarity loss based on a human annoted sentence similarity score.
+In terms of content, the samples are based on rather simple sentences.
+When the TSystems model was published, only the STSb dataset was used for STS training.
+It is also included in our model, but expanded to include SICK and Priya22 semantic textual relatedness:
+ - SICK was already used in parts in STSb, but our independent translation (XYZ) using DeepL leads to slightly different formulations. This approach allows more examples to be included in the training.
+ - The Priya22 semantic textual relatedness dataset published in 2022 was also translated into German via DeepL and added to the training data. Since it does not have a train test split, it was created independently at a ratio of 80:20.
+The rating scale of all datasets has been adjusted to STSb with a value range from 0 to 5.
+All training and test data (STSb, Sick, Priya22) were checked for duplicates within and with each other and removed if found.
+Because the test data has a higher priority, duplicated entries between test-train are always removed from train split.
+The final used datasets can be viewed here: XYZ.
+### Training
+Befor fine-tuning for STS we made the English paraphrasing model [paraphrase-distilroberta-base-v1](https://huggingface.co/sentence-transformers/paraphrase-distilroberta-base-v1) usable for German by applying Knowledge Distillation (Teacher-Student approach).
+The TSystems model used version 1, which is based on 7 different datasets and contains around 24.6 million samples.
+We are using version 2 with 12 datasets and about 83.3 million examples.
+Details for this process here: XYZ
+For fine-tuning we are using SBERT's [training_stsbenchmark_continue_training.py](https://github.com/UKPLab/sentence-transformers/blob/b86eec31cf0a102ad786ba1ff31bfeb4998d3ca5/examples/training/sts/training_stsbenchmark_continue_training.py) training script.
+One thing has been changed in this training script: when a sentence pair consists of identical utterances the score is set to 5.0 (maximum).
+It makes no sense to say identical sentences have a score of 4.8 or 4.9.
+#### Parameterization of training
+- **Script: [training_stsbenchmark_continue_training.py](https://github.com/UKPLab/sentence-transformers/blob/b86eec31cf0a102ad786ba1ff31bfeb4998d3ca5/examples/training/sts/training_stsbenchmark_continue_training.py)**
+- **Datasets: todo**
+- **GPU: NVIDIA A40 (Driver Version: 515.48.07; CUDA Version: 11.7)**
+- **Batch Size: 32**
+- **Base Model: todo**
+- **Loss Function: Cosine Similarity**
+- **Learning Rate: 2e-5**
+- **Epochs: 3**
+- **Evaluation Samples: 500**
+- **Evaluation Steps: 1000**
+- **Warmup Steps: 10%**
+- **:**
+### Evaluation <a name="evaluation"></a>
+ todo todo todo
+The evaluation is based on **[germanDPR](https://arxiv.org/abs/2104.12741)**.
+The dataset developed by [Deepset.ai](deepset.ai) consists of question-answer pairs, which are supplemented by three "hard negatives" per question.
+This makes it an ideal basis for benchmarking.
+Publicly available is the dataset as **[deepset/germanDPR](https://huggingface.co/datasets/deepset/germandpr)**, which does not support BEIR by default.
+Consequently, this dataset was also reworked manually.
+In addition, duplicate text elements were removed and minimal text adjustments were made.
+The details of this process can be found in **[PM-AI/germandpr-beir](https://huggingface.co/datasets/PM-AI/germandpr-beir)**.
+The BEIR-compatible germanDPR dataset consists of **9275 questions** with **23993 text passages** for the **train split**.
+In order to have enough text passages for information retrieval, we use the train split and not the test split.
+The following table shows the evaluation results for different approaches and models:
+**model**|**NDCG@1**|**NDCG@10**|**NDCG@100**|**comment**
+:-----:|:-----:|:-----:|:-----:|:-----:
+bi-encoder_msmarco_bert-base_german (new) | 0.5300 <br /> 🏆 | 0.7196 <br /> 🏆 | 0.7360 <br /> 🏆 | "OUR model"
+[deepset/gbert-base-germandpr-X_encoder](https://huggingface.co/deepset/gbert-base-germandpr-ctx_encoder) | 0.4828 | 0.6970 | 0.7147 | "has two encoder models (one for queries and one for corpus), is SOTA approach"
+[distiluse-base-multilingual-cased-v1](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v1) | 0.4561 | 0.6347 | 0.6613 | "trained on 15 languages"
+[paraphrase-multilingual-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-mpnet-base-v2) | 0.4511 | 0.6328 | 0.6592 | "trained on huge corpus, support for 50+ languages"
+[distiluse-base-multilingual-cased-v2](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v2) | 0.4350 | 0.6103 | 0.6411 | "trained on 50+ languages"
+[sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) | 0.4168 | 0.5931 | 0.6237 | "trained on large corpus, support for 50+ languages"
+[svalabs/bi-electra-ms-marco-german-uncased](svalabs/bi-electra-ms-marco-german-uncased) | 0.3818 | 0.5663 | 0.5986 | "most similar to OUR model"
+[BM25](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html#bm25) | 0.3196 | 0.5377 | 0.5740 | "lexical approach"
+**❗It is crucial to understand that the comparisons are also made with models based on other transformer approaches���**
+A direct comparison based on the same approach can be made with [svalabs/bi-electra-ms-marco-german-uncased](svalabs/bi-electra-ms-marco-german-uncased).
+In this case, the model presented here outperforms its predecessor by up to 14 percentage points.
+Comparing with [deepset/gbert-base-germandpr-X_encoder](https://huggingface.co/deepset/gbert-base-germandpr-ctx_encoder) is theoretically a little unfair since deepset's approach is based on two models at the same time!
+Queries and passages are encoded separately which leads to a better, more superior contextualization.
+Still, our newly trained model is outperforming the other approach by around two percentage points.
+In addition, using two models at the same time also increases demands on memory and CPU power which causes higher costs.
+This makes the approach presented here even more valuable.
+Note:
+- Texts used for evaluation are sometimes very long. All models, except for BM25 approach, truncate the incoming texts some point. This can decrease performance.
+- Evaluation of deepset's gbert-base-germandpr model might give an incorrect impression. The model was originally trained on the data we used for evaluation (not 1:1 but almost).
+### Acknowledgment
+This work is a collaboration between [Technical University of Applied Sciences Wildau (TH Wildau)](https://en.th-wildau.de/) and [sense.ai.tion GmbH](https://senseaition.com/).
+You can contact us via:
+* [Philipp Müller (M.Eng.)](https://www.linkedin.com/in/herrphilipps); Author
+* [Prof. Dr. Janett Mohnke](mailto:[email protected]); TH Wildau
+* [Dr. Matthias Boldt, Jörg Oehmichen](mailto:[email protected]); sense.AI.tion GmbH
+This work was funded by the European Regional Development Fund (EFRE) and the State of Brandenburg. Project/Vorhaben: "ProFIT: Natürlichsprachliche Dialogassistenten in der Pflege".
+<div style="display:flex">
+     <div style="padding-left:20px;">
+          <a href="https://efre.brandenburg.de/efre/de/"><img src="https://huggingface.co/datasets/PM-AI/germandpr-beir/resolve/main/res/EFRE-Logo_rechts_oweb_en_rgb.jpeg" alt="Logo of European Regional Development Fund (EFRE)" width="200"/></a>
+     </div>
+     <div style="padding-left:20px;">
+          <a href="https://www.senseaition.com"><img src="https://senseaition.com/wp-content/uploads/thegem-logos/logo_c847aaa8f42141c4055d4a8665eb208d_3x.png" alt="Logo of senseaition GmbH" width="200"/></a>
+     </div>
+     <div style="padding-left:20px;">
+          <a href="https://www.th-wildau.de"><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/f/f6/TH_Wildau_Logo.png/640px-TH_Wildau_Logo.png" alt="Logo of TH Wildau" width="180"/></a>
+     </div>
+</div>