SentenceTransformer based on google-bert/bert-base-multilingual-cased

This is a sentence-transformers model finetuned from google-bert/bert-base-multilingual-cased. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: google-bert/bert-base-multilingual-cased
  • Maximum Sequence Length: 128 tokens
  • Output Dimensionality: 768 dimensions
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("luanafelbarros/bert-base-multilingual-cased-matryoshka-mkqa")
# Run inference
sentences = [
    'who wrote the song i shot the sheriff',
    'i shot the sheriff şarkısını kim besteledi',
    'tr',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Knowledge Distillation

  • Datasets: MSE-val-en-to-ar, MSE-val-en-to-da, MSE-val-en-to-de, MSE-val-en-to-en, MSE-val-en-to-es, MSE-val-en-to-fi, MSE-val-en-to-fr, MSE-val-en-to-he, MSE-val-en-to-hu, MSE-val-en-to-it, MSE-val-en-to-ja, MSE-val-en-to-ko, MSE-val-en-to-km, MSE-val-en-to-ms, MSE-val-en-to-nl, MSE-val-en-to-no, MSE-val-en-to-pl, MSE-val-en-to-pt, MSE-val-en-to-ru, MSE-val-en-to-sv, MSE-val-en-to-th, MSE-val-en-to-tr, MSE-val-en-to-vi, MSE-val-en-to-zh_cn, MSE-val-en-to-zh_hk and MSE-val-en-to-zh_tw
  • Evaluated with MSEEvaluator
Metric MSE-val-en-to-ar MSE-val-en-to-da MSE-val-en-to-de MSE-val-en-to-en MSE-val-en-to-es MSE-val-en-to-fi MSE-val-en-to-fr MSE-val-en-to-he MSE-val-en-to-hu MSE-val-en-to-it MSE-val-en-to-ja MSE-val-en-to-ko MSE-val-en-to-km MSE-val-en-to-ms MSE-val-en-to-nl MSE-val-en-to-no MSE-val-en-to-pl MSE-val-en-to-pt MSE-val-en-to-ru MSE-val-en-to-sv MSE-val-en-to-th MSE-val-en-to-tr MSE-val-en-to-vi MSE-val-en-to-zh_cn MSE-val-en-to-zh_hk MSE-val-en-to-zh_tw
negative_mse -18.9326 -15.6858 -16.1256 -13.3884 -15.6481 -17.1741 -15.8143 -18.4839 -17.5854 -15.7066 -17.8007 -19.2666 -28.3875 -15.7831 -15.0272 -15.5984 -16.6414 -15.7691 -16.9116 -15.5558 -18.3703 -16.9459 -16.4828 -16.9964 -16.8207 -17.3817

Training Details

Training Dataset

Unnamed Dataset

  • Size: 234,000 training samples
  • Columns: english, non-english, target, and label
  • Approximate statistics based on the first 1000 samples:
    english non-english target label
    type string string string list
    details
    • min: 10 tokens
    • mean: 12.34 tokens
    • max: 18 tokens
    • min: 3 tokens
    • mean: 14.41 tokens
    • max: 49 tokens
    • min: 3 tokens
    • mean: 3.38 tokens
    • max: 7 tokens
    • size: 768 elements
  • Samples:
    english non-english target label
    who plays hope on days of our lives من الذي يلعب الأمل في أيام حياتنا ar [0.2171212136745453, 0.5138550996780396, 0.5517176389694214, -1.0655105113983154, 1.5853567123413086, ...]
    who plays hope on days of our lives hvem spiller hope i Horton-sagaen da [0.2171212136745453, 0.5138550996780396, 0.5517176389694214, -1.0655105113983154, 1.5853567123413086, ...]
    who plays hope on days of our lives Wer spielt die Hope in Zeit der Sehnsucht? de [0.2171212136745453, 0.5138550996780396, 0.5517176389694214, -1.0655105113983154, 1.5853567123413086, ...]
  • Loss: MSELoss

Evaluation Dataset

Unnamed Dataset

  • Size: 13,000 evaluation samples
  • Columns: english, non-english, target, and label
  • Approximate statistics based on the first 1000 samples:
    english non-english target label
    type string string string list
    details
    • min: 10 tokens
    • mean: 12.44 tokens
    • max: 16 tokens
    • min: 3 tokens
    • mean: 14.48 tokens
    • max: 49 tokens
    • min: 3 tokens
    • mean: 3.38 tokens
    • max: 7 tokens
    • size: 768 elements
  • Samples:
    english non-english target label
    who played prudence on nanny and the professor من لعب دور "prudence" فى "nanny and the professor" ar [-0.2837616801261902, -0.4943353235721588, 0.020107418298721313, 0.7796109318733215, -0.47365888953208923, ...]
    who played prudence on nanny and the professor hvem spiller prudence på nanny and the professor da [-0.2837616801261902, -0.4943353235721588, 0.020107418298721313, 0.7796109318733215, -0.47365888953208923, ...]
    who played prudence on nanny and the professor Wer spielte Prudence in Nanny and the Professor de [-0.2837616801261902, -0.4943353235721588, 0.020107418298721313, 0.7796109318733215, -0.47365888953208923, ...]
  • Loss: MSELoss

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 64
  • per_device_eval_batch_size: 64
  • learning_rate: 1e-05
  • num_train_epochs: 4
  • warmup_ratio: 0.1
  • fp16: True

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 64
  • per_device_eval_batch_size: 64
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 1e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 4
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss Validation Loss MSE-val-en-to-ar_negative_mse MSE-val-en-to-da_negative_mse MSE-val-en-to-de_negative_mse MSE-val-en-to-en_negative_mse MSE-val-en-to-es_negative_mse MSE-val-en-to-fi_negative_mse MSE-val-en-to-fr_negative_mse MSE-val-en-to-he_negative_mse MSE-val-en-to-hu_negative_mse MSE-val-en-to-it_negative_mse MSE-val-en-to-ja_negative_mse MSE-val-en-to-ko_negative_mse MSE-val-en-to-km_negative_mse MSE-val-en-to-ms_negative_mse MSE-val-en-to-nl_negative_mse MSE-val-en-to-no_negative_mse MSE-val-en-to-pl_negative_mse MSE-val-en-to-pt_negative_mse MSE-val-en-to-ru_negative_mse MSE-val-en-to-sv_negative_mse MSE-val-en-to-th_negative_mse MSE-val-en-to-tr_negative_mse MSE-val-en-to-vi_negative_mse MSE-val-en-to-zh_cn_negative_mse MSE-val-en-to-zh_hk_negative_mse MSE-val-en-to-zh_tw_negative_mse
0.1367 500 0.3783 - - - - - - - - - - - - - - - - - - - - - - - - - - -
0.2734 1000 0.3256 0.3071 -30.0050 -29.7152 -29.7584 -29.5204 -29.6875 -29.9032 -29.6918 -29.9795 -29.9430 -29.7142 -29.8220 -30.0745 -32.1218 -29.8042 -29.7132 -29.7625 -29.7677 -29.6658 -29.8250 -29.8242 -30.1233 -29.8640 -29.7497 -29.6833 -29.7296 -29.7063
0.4102 1500 0.3007 - - - - - - - - - - - - - - - - - - - - - - - - - - -
0.5469 2000 0.2795 0.2663 -25.0193 -23.8364 -23.9924 -22.8145 -23.7158 -24.4490 -23.7719 -24.6885 -24.5973 -23.7662 -24.4998 -25.3625 -30.9153 -24.0474 -23.5674 -23.7934 -24.1332 -23.6279 -24.1308 -23.8860 -25.4166 -24.4840 -24.1931 -24.0816 -24.0634 -24.2529
0.6836 2500 0.2659 - - - - - - - - - - - - - - - - - - - - - - - - - - -
0.8203 3000 0.2562 0.2487 -22.9862 -21.2544 -21.4573 -19.8714 -21.1251 -22.1884 -21.1984 -22.6963 -22.3069 -21.1959 -22.3180 -23.4410 -30.2373 -21.4324 -20.8799 -21.1834 -21.7427 -21.1291 -21.7291 -21.3003 -23.2994 -22.1537 -21.7480 -21.7521 -21.6844 -21.9702
0.9571 3500 0.2475 - - - - - - - - - - - - - - - - - - - - - - - - - - -
1.0938 4000 0.2411 0.2375 -21.8220 -19.6064 -19.9128 -17.9872 -19.5372 -20.7666 -19.6563 -21.4985 -20.9295 -19.6182 -20.9963 -22.2441 -29.7291 -19.8001 -19.2003 -19.5189 -20.2697 -19.5946 -20.3160 -19.6652 -21.9553 -20.6678 -20.2305 -20.3719 -20.2700 -20.6528
1.2305 4500 0.2351 - - - - - - - - - - - - - - - - - - - - - - - - - - -
1.3672 5000 0.23 0.2296 -21.0058 -18.4861 -18.7926 -16.6395 -18.4034 -19.7517 -18.5299 -20.6663 -19.9769 -18.4977 -20.0496 -21.4171 -29.3272 -18.6213 -17.9746 -18.3449 -19.2392 -18.4960 -19.3377 -18.5079 -20.9805 -19.5803 -19.1385 -19.4256 -19.2708 -19.7140
1.5040 5500 0.2257 - - - - - - - - - - - - - - - - - - - - - - - - - - -
1.6407 6000 0.2222 0.2245 -20.4317 -17.7592 -18.1037 -15.7487 -17.6947 -19.0287 -17.8518 -20.1401 -19.3864 -17.7539 -19.4615 -20.8562 -29.1081 -17.8707 -17.1892 -17.6230 -18.5879 -17.7857 -18.7075 -17.7347 -20.2941 -18.8814 -18.4449 -18.8036 -18.6146 -19.1169
1.7774 6500 0.2186 - - - - - - - - - - - - - - - - - - - - - - - - - - -
1.9141 7000 0.2158 0.2199 -19.9961 -17.0956 -17.4488 -14.9930 -17.0238 -18.4442 -17.1720 -19.6005 -18.7765 -17.1020 -18.8972 -20.3720 -28.8656 -17.1949 -16.4824 -16.9655 -17.9687 -17.1229 -18.0911 -17.0128 -19.6600 -18.2823 -17.8109 -18.2341 -18.0582 -18.5735
2.0509 7500 0.2135 - - - - - - - - - - - - - - - - - - - - - - - - - - -
2.1876 8000 0.2109 0.2167 -19.6376 -16.6362 -17.0307 -14.4461 -16.5766 -18.0419 -16.7080 -19.2403 -18.3971 -16.6443 -18.5251 -20.0263 -28.7414 -16.7279 -15.9992 -16.5092 -17.5170 -16.6766 -17.7151 -16.5403 -19.2861 -17.8316 -17.3764 -17.8453 -17.6606 -18.1844
2.3243 8500 0.2088 - - - - - - - - - - - - - - - - - - - - - - - - - - -
2.4610 9000 0.2074 0.2149 -19.4358 -16.3728 -16.7740 -14.1447 -16.3289 -17.8191 -16.4582 -19.0369 -18.1738 -16.3903 -18.3565 -19.8207 -28.6133 -16.4804 -15.7354 -16.2673 -17.3034 -16.4190 -17.4826 -16.2566 -18.9971 -17.5950 -17.1273 -17.6066 -17.4124 -17.9799
2.5978 9500 0.2059 - - - - - - - - - - - - - - - - - - - - - - - - - - -
2.7345 10000 0.2047 0.2134 -19.2764 -16.1718 -16.5449 -13.8928 -16.1098 -17.5866 -16.2421 -18.8665 -17.9798 -16.1538 -18.1695 -19.6218 -28.5605 -16.2479 -15.4962 -16.0522 -17.0797 -16.2106 -17.3130 -16.0278 -18.8206 -17.3910 -16.9231 -17.4203 -17.2266 -17.7903
2.8712 10500 0.2033 - - - - - - - - - - - - - - - - - - - - - - - - - - -
3.0079 11000 0.2024 0.2120 -19.1026 -15.9149 -16.3497 -13.6750 -15.8828 -17.3842 -16.0397 -18.6612 -17.7796 -15.9436 -17.9779 -19.4370 -28.4678 -16.0245 -15.2818 -15.8265 -16.8594 -15.9988 -17.1163 -15.8106 -18.5870 -17.1548 -16.7074 -17.2082 -17.0233 -17.5910
3.1447 11500 0.201 - - - - - - - - - - - - - - - - - - - - - - - - - - -
3.2814 12000 0.2004 0.2112 -19.0406 -15.8196 -16.2516 -13.5420 -15.7688 -17.2734 -15.9280 -18.5894 -17.6966 -15.8265 -17.8933 -19.3785 -28.4539 -15.9129 -15.1631 -15.7175 -16.7540 -15.8974 -17.0251 -15.6875 -18.4807 -17.0615 -16.6087 -17.1051 -16.9423 -17.4923
3.4181 12500 0.1997 - - - - - - - - - - - - - - - - - - - - - - - - - - -
3.5548 13000 0.1995 0.2108 -18.9779 -15.7524 -16.1996 -13.4723 -15.7211 -17.2272 -15.8790 -18.5412 -17.6416 -15.7862 -17.8502 -19.3124 -28.4179 -15.8513 -15.1030 -15.6645 -16.7053 -15.8355 -16.9742 -15.6246 -18.4384 -17.0053 -16.5478 -17.0674 -16.8851 -17.4527
3.6916 13500 0.1991 - - - - - - - - - - - - - - - - - - - - - - - - - - -
3.8283 14000 0.1987 0.2103 -18.9326 -15.6858 -16.1256 -13.3884 -15.6481 -17.1741 -15.8143 -18.4839 -17.5854 -15.7066 -17.8007 -19.2666 -28.3875 -15.7831 -15.0272 -15.5984 -16.6414 -15.7691 -16.9116 -15.5558 -18.3703 -16.9459 -16.4828 -16.9964 -16.8207 -17.3817
3.9650 14500 0.1989 - - - - - - - - - - - - - - - - - - - - - - - - - - -

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 3.3.1
  • Transformers: 4.46.3
  • PyTorch: 2.5.1+cu121
  • Accelerate: 1.1.1
  • Datasets: 3.1.0
  • Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MSELoss

@inproceedings{reimers-2020-multilingual-sentence-bert,
    title = "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2020",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2004.09813",
}
Downloads last month
9
Safetensors
Model size
178M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for luanafelbarros/bert-base-multilingual-cased-matryoshka-mkqa

Finetuned
(582)
this model

Evaluation results