diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -1,3 +1,9120 @@ ---- -license: mit ---- +--- +tags: +- mteb +- transformers + +model-index: +- name: Zeta-Alpha-E5-Mistral + results: + - dataset: + config: en-ext + name: MTEB AmazonCounterfactualClassification (en-ext) + revision: e8379541af4e31359cca9fbcf4b00f2671dba205 + split: test + type: mteb/amazon_counterfactual + metrics: + - type: accuracy + value: 78.61319340329835 + - type: ap + value: 26.407069359264725 + - type: ap_weighted + value: 26.407069359264725 + - type: f1 + value: 65.36726648428174 + - type: f1_weighted + value: 82.25843407936554 + - type: main_score + value: 78.61319340329835 + task: + type: Classification + - dataset: + config: en + name: MTEB AmazonCounterfactualClassification (en) + revision: e8379541af4e31359cca9fbcf4b00f2671dba205 + split: test + type: mteb/amazon_counterfactual + metrics: + - type: accuracy + value: 77.76119402985074 + - type: ap + value: 39.97673988468886 + - type: ap_weighted + value: 39.97673988468886 + - type: f1 + value: 71.23171737695898 + - type: f1_weighted + value: 79.55230307558237 + - type: main_score + value: 77.76119402985074 + task: + type: Classification + - dataset: + config: default + name: MTEB AmazonPolarityClassification (default) + revision: e2d317d38cd51312af73b3d32a06d1a08b442046 + split: test + type: mteb/amazon_polarity + metrics: + - type: accuracy + value: 96.61810000000001 + - type: ap + value: 94.99559013902017 + - type: ap_weighted + value: 94.99559013902017 + - type: f1 + value: 96.61758649480731 + - type: f1_weighted + value: 96.61758649480731 + - type: main_score + value: 96.61810000000001 + task: + type: Classification + - dataset: + config: en + name: MTEB AmazonReviewsClassification (en) + revision: 1399c76144fd37290681b995c656ef9b2e06e26d + split: test + type: mteb/amazon_reviews_multi + metrics: + - type: accuracy + value: 59.26199999999999 + - type: f1 + value: 56.32963321217333 + - type: f1_weighted + value: 56.32963321217333 + - type: main_score + value: 59.26199999999999 + task: + type: Classification + - dataset: + config: default + name: MTEB ArguAna (default) + revision: c22ab2a51041ffd869aaddef7af8d8215647e41a + split: test + type: mteb/arguana + metrics: + - type: main_score + value: 65.623 + - type: map_at_1 + value: 41.536 + - type: map_at_10 + value: 57.485 + - type: map_at_100 + value: 58.013000000000005 + - type: map_at_1000 + value: 58.013000000000005 + - type: map_at_20 + value: 57.957 + - type: map_at_3 + value: 53.284 + - type: map_at_5 + value: 55.837 + - type: mrr_at_1 + value: 42.17638691322902 + - type: mrr_at_10 + value: 57.7096175122492 + - type: mrr_at_100 + value: 58.23610809196743 + - type: mrr_at_1000 + value: 58.23673750573145 + - type: mrr_at_20 + value: 58.180348622747324 + - type: mrr_at_3 + value: 53.44950213371275 + - type: mrr_at_5 + value: 56.07396870554779 + - type: nauc_map_at_1000_diff1 + value: 14.098091070036958 + - type: nauc_map_at_1000_max + value: -16.568377380844108 + - type: nauc_map_at_1000_std + value: -22.105696179585834 + - type: nauc_map_at_100_diff1 + value: 14.096542453201625 + - type: nauc_map_at_100_max + value: -16.57054590195526 + - type: nauc_map_at_100_std + value: -22.1090324366772 + - type: nauc_map_at_10_diff1 + value: 13.840246695558884 + - type: nauc_map_at_10_max + value: -16.52098795923224 + - type: nauc_map_at_10_std + value: -22.074328710004032 + - type: nauc_map_at_1_diff1 + value: 17.117727049808984 + - type: nauc_map_at_1_max + value: -18.587242049712614 + - type: nauc_map_at_1_std + value: -22.454707653498595 + - type: nauc_map_at_20_diff1 + value: 14.068130846454585 + - type: nauc_map_at_20_max + value: -16.53942858114966 + - type: nauc_map_at_20_std + value: -22.10921004077996 + - type: nauc_map_at_3_diff1 + value: 14.596579595737097 + - type: nauc_map_at_3_max + value: -15.62887067464894 + - type: nauc_map_at_3_std + value: -22.09058102549274 + - type: nauc_map_at_5_diff1 + value: 13.798507062284514 + - type: nauc_map_at_5_max + value: -16.36834850771522 + - type: nauc_map_at_5_std + value: -21.984206595455134 + - type: nauc_mrr_at_1000_diff1 + value: 12.144909427474602 + - type: nauc_mrr_at_1000_max + value: -17.048787138426324 + - type: nauc_mrr_at_1000_std + value: -21.961966140898564 + - type: nauc_mrr_at_100_diff1 + value: 12.143403633612827 + - type: nauc_mrr_at_100_max + value: -17.050945262411012 + - type: nauc_mrr_at_100_std + value: -21.965305811191673 + - type: nauc_mrr_at_10_diff1 + value: 11.88548720648553 + - type: nauc_mrr_at_10_max + value: -16.996705857584736 + - type: nauc_mrr_at_10_std + value: -21.883645748542396 + - type: nauc_mrr_at_1_diff1 + value: 15.37682964765565 + - type: nauc_mrr_at_1_max + value: -17.989361001169087 + - type: nauc_mrr_at_1_std + value: -21.697830490637955 + - type: nauc_mrr_at_20_diff1 + value: 12.119044499779363 + - type: nauc_mrr_at_20_max + value: -17.018675761117027 + - type: nauc_mrr_at_20_std + value: -21.965554459307565 + - type: nauc_mrr_at_3_diff1 + value: 12.535001807278187 + - type: nauc_mrr_at_3_max + value: -16.38816957172248 + - type: nauc_mrr_at_3_std + value: -22.081293367465896 + - type: nauc_mrr_at_5_diff1 + value: 11.892111947679496 + - type: nauc_mrr_at_5_max + value: -16.79709351116846 + - type: nauc_mrr_at_5_std + value: -21.79512696140714 + - type: nauc_ndcg_at_1000_diff1 + value: 13.67006999549869 + - type: nauc_ndcg_at_1000_max + value: -16.236125687432107 + - type: nauc_ndcg_at_1000_std + value: -21.810131960233065 + - type: nauc_ndcg_at_100_diff1 + value: 13.637478389163462 + - type: nauc_ndcg_at_100_max + value: -16.28219720987127 + - type: nauc_ndcg_at_100_std + value: -21.880912370176876 + - type: nauc_ndcg_at_10_diff1 + value: 12.558591199280556 + - type: nauc_ndcg_at_10_max + value: -15.952826009827106 + - type: nauc_ndcg_at_10_std + value: -21.818643731025382 + - type: nauc_ndcg_at_1_diff1 + value: 17.117727049808984 + - type: nauc_ndcg_at_1_max + value: -18.587242049712614 + - type: nauc_ndcg_at_1_std + value: -22.454707653498595 + - type: nauc_ndcg_at_20_diff1 + value: 13.402986057386181 + - type: nauc_ndcg_at_20_max + value: -16.072631062968746 + - type: nauc_ndcg_at_20_std + value: -21.98468803430586 + - type: nauc_ndcg_at_3_diff1 + value: 14.059904782033348 + - type: nauc_ndcg_at_3_max + value: -14.433190101994514 + - type: nauc_ndcg_at_3_std + value: -21.990025270634135 + - type: nauc_ndcg_at_5_diff1 + value: 12.434165121057134 + - type: nauc_ndcg_at_5_max + value: -15.650774158031522 + - type: nauc_ndcg_at_5_std + value: -21.636716447934305 + - type: nauc_precision_at_1000_diff1 + value: 1.7151819945276745 + - type: nauc_precision_at_1000_max + value: 20.85546049013785 + - type: nauc_precision_at_1000_std + value: 77.3551884133584 + - type: nauc_precision_at_100_diff1 + value: -7.961881099019577 + - type: nauc_precision_at_100_max + value: -1.8225484680865736 + - type: nauc_precision_at_100_std + value: 35.484449109425384 + - type: nauc_precision_at_10_diff1 + value: 0.46638305609538855 + - type: nauc_precision_at_10_max + value: -11.023993018739485 + - type: nauc_precision_at_10_std + value: -19.111584616037852 + - type: nauc_precision_at_1_diff1 + value: 17.117727049808984 + - type: nauc_precision_at_1_max + value: -18.587242049712614 + - type: nauc_precision_at_1_std + value: -22.454707653498595 + - type: nauc_precision_at_20_diff1 + value: -1.0298881487766305 + - type: nauc_precision_at_20_max + value: -4.548017977674335 + - type: nauc_precision_at_20_std + value: -18.901496352112133 + - type: nauc_precision_at_3_diff1 + value: 12.350178962124566 + - type: nauc_precision_at_3_max + value: -10.271126387937858 + - type: nauc_precision_at_3_std + value: -21.655307623793433 + - type: nauc_precision_at_5_diff1 + value: 6.011571432832696 + - type: nauc_precision_at_5_max + value: -12.478026665421389 + - type: nauc_precision_at_5_std + value: -19.845124181363882 + - type: nauc_recall_at_1000_diff1 + value: 1.7151819945236155 + - type: nauc_recall_at_1000_max + value: 20.855460490135933 + - type: nauc_recall_at_1000_std + value: 77.35518841335626 + - type: nauc_recall_at_100_diff1 + value: -7.961881099020542 + - type: nauc_recall_at_100_max + value: -1.8225484680932273 + - type: nauc_recall_at_100_std + value: 35.48444910942399 + - type: nauc_recall_at_10_diff1 + value: 0.46638305609538805 + - type: nauc_recall_at_10_max + value: -11.023993018739322 + - type: nauc_recall_at_10_std + value: -19.11158461603798 + - type: nauc_recall_at_1_diff1 + value: 17.117727049808984 + - type: nauc_recall_at_1_max + value: -18.587242049712614 + - type: nauc_recall_at_1_std + value: -22.454707653498595 + - type: nauc_recall_at_20_diff1 + value: -1.029888148776229 + - type: nauc_recall_at_20_max + value: -4.548017977673906 + - type: nauc_recall_at_20_std + value: -18.901496352110804 + - type: nauc_recall_at_3_diff1 + value: 12.350178962124682 + - type: nauc_recall_at_3_max + value: -10.271126387937805 + - type: nauc_recall_at_3_std + value: -21.65530762379337 + - type: nauc_recall_at_5_diff1 + value: 6.0115714328326435 + - type: nauc_recall_at_5_max + value: -12.478026665421405 + - type: nauc_recall_at_5_std + value: -19.845124181363875 + - type: ndcg_at_1 + value: 41.536 + - type: ndcg_at_10 + value: 65.623 + - type: ndcg_at_100 + value: 67.63 + - type: ndcg_at_1000 + value: 67.64099999999999 + - type: ndcg_at_20 + value: 67.241 + - type: ndcg_at_3 + value: 57.048 + - type: ndcg_at_5 + value: 61.678999999999995 + - type: precision_at_1 + value: 41.536 + - type: precision_at_10 + value: 9.132 + - type: precision_at_100 + value: 0.996 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 4.8759999999999994 + - type: precision_at_3 + value: 22.641 + - type: precision_at_5 + value: 15.845999999999998 + - type: recall_at_1 + value: 41.536 + - type: recall_at_10 + value: 91.323 + - type: recall_at_100 + value: 99.57300000000001 + - type: recall_at_1000 + value: 99.644 + - type: recall_at_20 + value: 97.51100000000001 + - type: recall_at_3 + value: 67.923 + - type: recall_at_5 + value: 79.232 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ArxivClusteringP2P (default) + revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d + split: test + type: mteb/arxiv-clustering-p2p + metrics: + - type: main_score + value: 52.026635060007244 + - type: v_measure + value: 52.026635060007244 + - type: v_measure_std + value: 14.357137408692006 + task: + type: Clustering + - dataset: + config: default + name: MTEB ArxivClusteringS2S (default) + revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 + split: test + type: mteb/arxiv-clustering-s2s + metrics: + - type: main_score + value: 47.834914950269855 + - type: v_measure + value: 47.834914950269855 + - type: v_measure_std + value: 14.487028918517247 + task: + type: Clustering + - dataset: + config: default + name: MTEB AskUbuntuDupQuestions (default) + revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 + split: test + type: mteb/askubuntudupquestions-reranking + metrics: + - type: main_score + value: 64.5808745066313 + - type: map + value: 64.5808745066313 + - type: mrr + value: 77.56540473991997 + - type: nAUC_map_diff1 + value: 23.168800779252464 + - type: nAUC_map_max + value: 30.342203769599735 + - type: nAUC_map_std + value: 22.562701982176833 + - type: nAUC_mrr_diff1 + value: 27.79261544540621 + - type: nAUC_mrr_max + value: 43.302228243606045 + - type: nAUC_mrr_std + value: 26.432985515912673 + task: + type: Reranking + - dataset: + config: default + name: MTEB BIOSSES (default) + revision: d3fb88f8f02e40887cd149695127462bbcf29b4a + split: test + type: mteb/biosses-sts + metrics: + - type: cosine_pearson + value: 87.97778619889539 + - type: cosine_spearman + value: 86.44233109293758 + - type: euclidean_pearson + value: 86.6664224630525 + - type: euclidean_spearman + value: 86.44233109293758 + - type: main_score + value: 86.44233109293758 + - type: manhattan_pearson + value: 86.75174487553707 + - type: manhattan_spearman + value: 86.61402175201368 + - type: pearson + value: 87.97778619889539 + - type: spearman + value: 86.44233109293758 + task: + type: STS + - dataset: + config: default + name: MTEB Banking77Classification (default) + revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 + split: test + type: mteb/banking77 + metrics: + - type: accuracy + value: 83.1103896103896 + - type: f1 + value: 82.2932953112279 + - type: f1_weighted + value: 82.2932953112279 + - type: main_score + value: 83.1103896103896 + task: + type: Classification + - dataset: + config: default + name: MTEB BiorxivClusteringP2P (default) + revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 + split: test + type: mteb/biorxiv-clustering-p2p + metrics: + - type: main_score + value: 43.73746639290943 + - type: v_measure + value: 43.73746639290943 + - type: v_measure_std + value: 0.8808902310879784 + task: + type: Clustering + - dataset: + config: default + name: MTEB BiorxivClusteringS2S (default) + revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 + split: test + type: mteb/biorxiv-clustering-s2s + metrics: + - type: main_score + value: 40.73737737926463 + - type: v_measure + value: 40.73737737926463 + - type: v_measure_std + value: 0.6059982328960863 + task: + type: Clustering + - dataset: + config: default + name: MTEB CQADupstackAndroidRetrieval (default) + revision: f46a197baaae43b4f621051089b82a364682dfeb + split: test + type: mteb/cqadupstack-android + metrics: + - type: main_score + value: 57.92699999999999 + - type: map_at_1 + value: 37.622 + - type: map_at_10 + value: 50.722 + - type: map_at_100 + value: 52.352 + - type: map_at_1000 + value: 52.449999999999996 + - type: map_at_20 + value: 51.721 + - type: map_at_3 + value: 46.372 + - type: map_at_5 + value: 48.617 + - type: mrr_at_1 + value: 46.49499284692418 + - type: mrr_at_10 + value: 57.111633399187035 + - type: mrr_at_100 + value: 57.74896548960492 + - type: mrr_at_1000 + value: 57.770289975528044 + - type: mrr_at_20 + value: 57.515755458443365 + - type: mrr_at_3 + value: 54.48259418216498 + - type: mrr_at_5 + value: 55.920362422508305 + - type: nauc_map_at_1000_diff1 + value: 53.17059261368312 + - type: nauc_map_at_1000_max + value: 41.5046142660243 + - type: nauc_map_at_1000_std + value: -11.356308313633614 + - type: nauc_map_at_100_diff1 + value: 53.199691716280284 + - type: nauc_map_at_100_max + value: 41.498598817616696 + - type: nauc_map_at_100_std + value: -11.300115349820244 + - type: nauc_map_at_10_diff1 + value: 53.12949956289881 + - type: nauc_map_at_10_max + value: 41.24444583290451 + - type: nauc_map_at_10_std + value: -12.268596847396134 + - type: nauc_map_at_1_diff1 + value: 57.0093058760594 + - type: nauc_map_at_1_max + value: 37.16324129131443 + - type: nauc_map_at_1_std + value: -12.618771182632155 + - type: nauc_map_at_20_diff1 + value: 53.09753260703357 + - type: nauc_map_at_20_max + value: 41.279196285215455 + - type: nauc_map_at_20_std + value: -11.602709303039035 + - type: nauc_map_at_3_diff1 + value: 53.60320552270635 + - type: nauc_map_at_3_max + value: 40.52972305912495 + - type: nauc_map_at_3_std + value: -13.28611477898908 + - type: nauc_map_at_5_diff1 + value: 53.27582851039211 + - type: nauc_map_at_5_max + value: 40.964556738286696 + - type: nauc_map_at_5_std + value: -12.994624672570426 + - type: nauc_mrr_at_1000_diff1 + value: 52.06292528781184 + - type: nauc_mrr_at_1000_max + value: 42.753551647067844 + - type: nauc_mrr_at_1000_std + value: -11.39896283255964 + - type: nauc_mrr_at_100_diff1 + value: 52.06462455470101 + - type: nauc_mrr_at_100_max + value: 42.74984437909525 + - type: nauc_mrr_at_100_std + value: -11.398086835953704 + - type: nauc_mrr_at_10_diff1 + value: 51.88682778785948 + - type: nauc_mrr_at_10_max + value: 42.758770739865916 + - type: nauc_mrr_at_10_std + value: -11.645879420520737 + - type: nauc_mrr_at_1_diff1 + value: 55.229276125235195 + - type: nauc_mrr_at_1_max + value: 42.59561790724058 + - type: nauc_mrr_at_1_std + value: -11.012331834858617 + - type: nauc_mrr_at_20_diff1 + value: 52.01322772146113 + - type: nauc_mrr_at_20_max + value: 42.753139349619005 + - type: nauc_mrr_at_20_std + value: -11.383800469326204 + - type: nauc_mrr_at_3_diff1 + value: 52.38756527449288 + - type: nauc_mrr_at_3_max + value: 43.20102042864951 + - type: nauc_mrr_at_3_std + value: -12.194697125093237 + - type: nauc_mrr_at_5_diff1 + value: 51.7152150896939 + - type: nauc_mrr_at_5_max + value: 42.68388976511927 + - type: nauc_mrr_at_5_std + value: -11.785435820434257 + - type: nauc_ndcg_at_1000_diff1 + value: 52.190748942344136 + - type: nauc_ndcg_at_1000_max + value: 42.289561613110926 + - type: nauc_ndcg_at_1000_std + value: -9.772840486820911 + - type: nauc_ndcg_at_100_diff1 + value: 52.28390241403799 + - type: nauc_ndcg_at_100_max + value: 42.16017073586535 + - type: nauc_ndcg_at_100_std + value: -9.06906927424865 + - type: nauc_ndcg_at_10_diff1 + value: 50.97819135053925 + - type: nauc_ndcg_at_10_max + value: 41.46589951311915 + - type: nauc_ndcg_at_10_std + value: -11.80310917598558 + - type: nauc_ndcg_at_1_diff1 + value: 55.229276125235195 + - type: nauc_ndcg_at_1_max + value: 42.59561790724058 + - type: nauc_ndcg_at_1_std + value: -11.012331834858617 + - type: nauc_ndcg_at_20_diff1 + value: 51.41064846134139 + - type: nauc_ndcg_at_20_max + value: 41.605248141412844 + - type: nauc_ndcg_at_20_std + value: -10.083386198861698 + - type: nauc_ndcg_at_3_diff1 + value: 51.458633181223824 + - type: nauc_ndcg_at_3_max + value: 41.96885472214017 + - type: nauc_ndcg_at_3_std + value: -12.200744805163257 + - type: nauc_ndcg_at_5_diff1 + value: 50.72067871530336 + - type: nauc_ndcg_at_5_max + value: 41.44401252353067 + - type: nauc_ndcg_at_5_std + value: -12.43883398550311 + - type: nauc_precision_at_1000_diff1 + value: -23.3655222272241 + - type: nauc_precision_at_1000_max + value: -10.673098129694008 + - type: nauc_precision_at_1000_std + value: -2.2922267459919192 + - type: nauc_precision_at_100_diff1 + value: -16.503526271943155 + - type: nauc_precision_at_100_max + value: -1.4997938185424997 + - type: nauc_precision_at_100_std + value: 8.71880738236966 + - type: nauc_precision_at_10_diff1 + value: 2.5128648280873356 + - type: nauc_precision_at_10_max + value: 17.467618699199885 + - type: nauc_precision_at_10_std + value: 4.796850983515245 + - type: nauc_precision_at_1_diff1 + value: 55.229276125235195 + - type: nauc_precision_at_1_max + value: 42.59561790724058 + - type: nauc_precision_at_1_std + value: -11.012331834858617 + - type: nauc_precision_at_20_diff1 + value: -6.407914358668061 + - type: nauc_precision_at_20_max + value: 9.426001961474876 + - type: nauc_precision_at_20_std + value: 10.259579239699693 + - type: nauc_precision_at_3_diff1 + value: 27.28776882706687 + - type: nauc_precision_at_3_max + value: 34.18365881158305 + - type: nauc_precision_at_3_std + value: -6.21559400162483 + - type: nauc_precision_at_5_diff1 + value: 15.118931845648875 + - type: nauc_precision_at_5_max + value: 27.58514180836989 + - type: nauc_precision_at_5_std + value: -1.2430569318682803 + - type: nauc_recall_at_1000_diff1 + value: 49.81611176181191 + - type: nauc_recall_at_1000_max + value: 55.2140058052434 + - type: nauc_recall_at_1000_std + value: 50.66546399222259 + - type: nauc_recall_at_100_diff1 + value: 47.637097806952546 + - type: nauc_recall_at_100_max + value: 39.927761652819996 + - type: nauc_recall_at_100_std + value: 18.699209483449884 + - type: nauc_recall_at_10_diff1 + value: 43.67415650636362 + - type: nauc_recall_at_10_max + value: 36.65890264338029 + - type: nauc_recall_at_10_std + value: -10.586842496609806 + - type: nauc_recall_at_1_diff1 + value: 57.0093058760594 + - type: nauc_recall_at_1_max + value: 37.16324129131443 + - type: nauc_recall_at_1_std + value: -12.618771182632155 + - type: nauc_recall_at_20_diff1 + value: 43.77443203371046 + - type: nauc_recall_at_20_max + value: 37.59402918351545 + - type: nauc_recall_at_20_std + value: -1.088884219935537 + - type: nauc_recall_at_3_diff1 + value: 47.54143108850364 + - type: nauc_recall_at_3_max + value: 38.721237478590986 + - type: nauc_recall_at_3_std + value: -12.934093993227872 + - type: nauc_recall_at_5_diff1 + value: 45.16027835802349 + - type: nauc_recall_at_5_max + value: 37.541928394402 + - type: nauc_recall_at_5_std + value: -12.7024498269252 + - type: ndcg_at_1 + value: 46.495 + - type: ndcg_at_10 + value: 57.92699999999999 + - type: ndcg_at_100 + value: 62.871 + - type: ndcg_at_1000 + value: 63.979 + - type: ndcg_at_20 + value: 60.123000000000005 + - type: ndcg_at_3 + value: 52.146 + - type: ndcg_at_5 + value: 54.592 + - type: precision_at_1 + value: 46.495 + - type: precision_at_10 + value: 11.302 + - type: precision_at_100 + value: 1.717 + - type: precision_at_1000 + value: 0.209 + - type: precision_at_20 + value: 6.694999999999999 + - type: precision_at_3 + value: 25.130999999999997 + - type: precision_at_5 + value: 18.112000000000002 + - type: recall_at_1 + value: 37.622 + - type: recall_at_10 + value: 71.29100000000001 + - type: recall_at_100 + value: 91.618 + - type: recall_at_1000 + value: 98.084 + - type: recall_at_20 + value: 79.06800000000001 + - type: recall_at_3 + value: 54.291 + - type: recall_at_5 + value: 61.277 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackEnglishRetrieval (default) + revision: ad9991cb51e31e31e430383c75ffb2885547b5f0 + split: test + type: mteb/cqadupstack-english + metrics: + - type: main_score + value: 54.81 + - type: map_at_1 + value: 36.318 + - type: map_at_10 + value: 48.604 + - type: map_at_100 + value: 49.957 + - type: map_at_1000 + value: 50.090999999999994 + - type: map_at_20 + value: 49.323 + - type: map_at_3 + value: 45.327 + - type: map_at_5 + value: 47.195 + - type: mrr_at_1 + value: 45.54140127388535 + - type: mrr_at_10 + value: 54.75361439692653 + - type: mrr_at_100 + value: 55.3186658675232 + - type: mrr_at_1000 + value: 55.35138313175343 + - type: mrr_at_20 + value: 55.090707932347385 + - type: mrr_at_3 + value: 52.547770700636995 + - type: mrr_at_5 + value: 53.824840764331285 + - type: nauc_map_at_1000_diff1 + value: 57.35593106002006 + - type: nauc_map_at_1000_max + value: 40.552729522973685 + - type: nauc_map_at_1000_std + value: -10.989177079656951 + - type: nauc_map_at_100_diff1 + value: 57.32052168334852 + - type: nauc_map_at_100_max + value: 40.46070210146782 + - type: nauc_map_at_100_std + value: -11.108868859892485 + - type: nauc_map_at_10_diff1 + value: 57.43556212755914 + - type: nauc_map_at_10_max + value: 39.602569239800914 + - type: nauc_map_at_10_std + value: -12.457373890189123 + - type: nauc_map_at_1_diff1 + value: 61.51923108672362 + - type: nauc_map_at_1_max + value: 33.66301245608268 + - type: nauc_map_at_1_std + value: -14.910296546906638 + - type: nauc_map_at_20_diff1 + value: 57.32450639339468 + - type: nauc_map_at_20_max + value: 40.00279666455951 + - type: nauc_map_at_20_std + value: -11.954877167188718 + - type: nauc_map_at_3_diff1 + value: 58.292976288976284 + - type: nauc_map_at_3_max + value: 38.990438822490226 + - type: nauc_map_at_3_std + value: -13.970657252411506 + - type: nauc_map_at_5_diff1 + value: 57.74458705936615 + - type: nauc_map_at_5_max + value: 39.262962520045605 + - type: nauc_map_at_5_std + value: -13.496132843620117 + - type: nauc_mrr_at_1000_diff1 + value: 55.884394510783494 + - type: nauc_mrr_at_1000_max + value: 43.08107315741987 + - type: nauc_mrr_at_1000_std + value: -6.724057429293513 + - type: nauc_mrr_at_100_diff1 + value: 55.864604240754645 + - type: nauc_mrr_at_100_max + value: 43.06677965639483 + - type: nauc_mrr_at_100_std + value: -6.72580749173002 + - type: nauc_mrr_at_10_diff1 + value: 55.90780608185463 + - type: nauc_mrr_at_10_max + value: 43.06061192523088 + - type: nauc_mrr_at_10_std + value: -6.8987326879790745 + - type: nauc_mrr_at_1_diff1 + value: 59.62713974738764 + - type: nauc_mrr_at_1_max + value: 42.84809412590511 + - type: nauc_mrr_at_1_std + value: -8.192730812780214 + - type: nauc_mrr_at_20_diff1 + value: 55.88887224383142 + - type: nauc_mrr_at_20_max + value: 43.107603437889836 + - type: nauc_mrr_at_20_std + value: -6.836935860117027 + - type: nauc_mrr_at_3_diff1 + value: 56.39427235586623 + - type: nauc_mrr_at_3_max + value: 43.462181668167474 + - type: nauc_mrr_at_3_std + value: -7.42603493425245 + - type: nauc_mrr_at_5_diff1 + value: 56.2072645636557 + - type: nauc_mrr_at_5_max + value: 43.41071339315302 + - type: nauc_mrr_at_5_std + value: -7.023538993321608 + - type: nauc_ndcg_at_1000_diff1 + value: 55.245918170289535 + - type: nauc_ndcg_at_1000_max + value: 41.881610576866976 + - type: nauc_ndcg_at_1000_std + value: -6.3371441769168 + - type: nauc_ndcg_at_100_diff1 + value: 54.920715029162196 + - type: nauc_ndcg_at_100_max + value: 41.356321193123904 + - type: nauc_ndcg_at_100_std + value: -6.852733612141818 + - type: nauc_ndcg_at_10_diff1 + value: 55.62709600161367 + - type: nauc_ndcg_at_10_max + value: 40.84954016457469 + - type: nauc_ndcg_at_10_std + value: -9.325118694396432 + - type: nauc_ndcg_at_1_diff1 + value: 59.62713974738764 + - type: nauc_ndcg_at_1_max + value: 42.84809412590511 + - type: nauc_ndcg_at_1_std + value: -8.192730812780214 + - type: nauc_ndcg_at_20_diff1 + value: 55.448359410036765 + - type: nauc_ndcg_at_20_max + value: 41.150252947798336 + - type: nauc_ndcg_at_20_std + value: -8.898385836525073 + - type: nauc_ndcg_at_3_diff1 + value: 55.98668423895621 + - type: nauc_ndcg_at_3_max + value: 42.16722172496642 + - type: nauc_ndcg_at_3_std + value: -9.596196418336083 + - type: nauc_ndcg_at_5_diff1 + value: 56.02917078591471 + - type: nauc_ndcg_at_5_max + value: 41.304861496815846 + - type: nauc_ndcg_at_5_std + value: -10.220505778783059 + - type: nauc_precision_at_1000_diff1 + value: -15.067849626234494 + - type: nauc_precision_at_1000_max + value: 10.711033593502213 + - type: nauc_precision_at_1000_std + value: 26.77332298698984 + - type: nauc_precision_at_100_diff1 + value: -8.615078069436368 + - type: nauc_precision_at_100_max + value: 18.3145596666278 + - type: nauc_precision_at_100_std + value: 30.21015663233885 + - type: nauc_precision_at_10_diff1 + value: 7.793876868365025 + - type: nauc_precision_at_10_max + value: 27.347816646420565 + - type: nauc_precision_at_10_std + value: 15.80899397311083 + - type: nauc_precision_at_1_diff1 + value: 59.62713974738764 + - type: nauc_precision_at_1_max + value: 42.84809412590511 + - type: nauc_precision_at_1_std + value: -8.192730812780214 + - type: nauc_precision_at_20_diff1 + value: 0.24864368395086062 + - type: nauc_precision_at_20_max + value: 23.832972766912174 + - type: nauc_precision_at_20_std + value: 20.786523005871533 + - type: nauc_precision_at_3_diff1 + value: 27.637060807440644 + - type: nauc_precision_at_3_max + value: 38.75856472426437 + - type: nauc_precision_at_3_std + value: 4.526960457398025 + - type: nauc_precision_at_5_diff1 + value: 17.426725059010508 + - type: nauc_precision_at_5_max + value: 32.91657967582915 + - type: nauc_precision_at_5_std + value: 8.659031122660748 + - type: nauc_recall_at_1000_diff1 + value: 43.99110429230081 + - type: nauc_recall_at_1000_max + value: 43.50760320081781 + - type: nauc_recall_at_1000_std + value: 37.35488908028419 + - type: nauc_recall_at_100_diff1 + value: 40.85961174692666 + - type: nauc_recall_at_100_max + value: 35.437032048092235 + - type: nauc_recall_at_100_std + value: 10.206380553502362 + - type: nauc_recall_at_10_diff1 + value: 47.963459727987484 + - type: nauc_recall_at_10_max + value: 35.321796597976146 + - type: nauc_recall_at_10_std + value: -8.271826590104487 + - type: nauc_recall_at_1_diff1 + value: 61.51923108672362 + - type: nauc_recall_at_1_max + value: 33.66301245608268 + - type: nauc_recall_at_1_std + value: -14.910296546906638 + - type: nauc_recall_at_20_diff1 + value: 46.0923078798864 + - type: nauc_recall_at_20_max + value: 36.231459192676255 + - type: nauc_recall_at_20_std + value: -5.842910208912178 + - type: nauc_recall_at_3_diff1 + value: 53.4247598566265 + - type: nauc_recall_at_3_max + value: 37.4811297920946 + - type: nauc_recall_at_3_std + value: -13.780739565588348 + - type: nauc_recall_at_5_diff1 + value: 51.42582720415568 + - type: nauc_recall_at_5_max + value: 36.371587572400216 + - type: nauc_recall_at_5_std + value: -12.571342054783571 + - type: ndcg_at_1 + value: 45.541 + - type: ndcg_at_10 + value: 54.81 + - type: ndcg_at_100 + value: 58.989000000000004 + - type: ndcg_at_1000 + value: 60.711000000000006 + - type: ndcg_at_20 + value: 56.433 + - type: ndcg_at_3 + value: 50.531000000000006 + - type: ndcg_at_5 + value: 52.549 + - type: precision_at_1 + value: 45.541 + - type: precision_at_10 + value: 10.395 + - type: precision_at_100 + value: 1.601 + - type: precision_at_1000 + value: 0.207 + - type: precision_at_20 + value: 6.045 + - type: precision_at_3 + value: 24.756 + - type: precision_at_5 + value: 17.413999999999998 + - type: recall_at_1 + value: 36.318 + - type: recall_at_10 + value: 65.553 + - type: recall_at_100 + value: 83.035 + - type: recall_at_1000 + value: 93.068 + - type: recall_at_20 + value: 71.443 + - type: recall_at_3 + value: 52.339999999999996 + - type: recall_at_5 + value: 58.225 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackGamingRetrieval (default) + revision: 4885aa143210c98657558c04aaf3dc47cfb54340 + split: test + type: mteb/cqadupstack-gaming + metrics: + - type: main_score + value: 66.25 + - type: map_at_1 + value: 46.717999999999996 + - type: map_at_10 + value: 60.245000000000005 + - type: map_at_100 + value: 61.228 + - type: map_at_1000 + value: 61.26200000000001 + - type: map_at_20 + value: 60.834999999999994 + - type: map_at_3 + value: 56.692 + - type: map_at_5 + value: 58.729 + - type: mrr_at_1 + value: 53.47962382445141 + - type: mrr_at_10 + value: 63.73092998955075 + - type: mrr_at_100 + value: 64.29853184715043 + - type: mrr_at_1000 + value: 64.31257152043639 + - type: mrr_at_20 + value: 64.10227012765188 + - type: mrr_at_3 + value: 61.536050156739876 + - type: mrr_at_5 + value: 62.76802507837003 + - type: nauc_map_at_1000_diff1 + value: 55.047095676710065 + - type: nauc_map_at_1000_max + value: 40.837917617105305 + - type: nauc_map_at_1000_std + value: -3.3525459727458866 + - type: nauc_map_at_100_diff1 + value: 55.03349307629215 + - type: nauc_map_at_100_max + value: 40.81920823558061 + - type: nauc_map_at_100_std + value: -3.3704449526194837 + - type: nauc_map_at_10_diff1 + value: 54.8896518211945 + - type: nauc_map_at_10_max + value: 40.273729782847255 + - type: nauc_map_at_10_std + value: -4.322391497542583 + - type: nauc_map_at_1_diff1 + value: 56.56780342832046 + - type: nauc_map_at_1_max + value: 32.052791522050285 + - type: nauc_map_at_1_std + value: -6.970554698390048 + - type: nauc_map_at_20_diff1 + value: 54.961536096551036 + - type: nauc_map_at_20_max + value: 40.73354880109598 + - type: nauc_map_at_20_std + value: -3.49610913844766 + - type: nauc_map_at_3_diff1 + value: 55.12266358852137 + - type: nauc_map_at_3_max + value: 38.45307401083733 + - type: nauc_map_at_3_std + value: -6.189594882006102 + - type: nauc_map_at_5_diff1 + value: 55.06081254258659 + - type: nauc_map_at_5_max + value: 39.44813045518199 + - type: nauc_map_at_5_std + value: -5.170534592297574 + - type: nauc_mrr_at_1000_diff1 + value: 55.62979777108712 + - type: nauc_mrr_at_1000_max + value: 42.673296693557006 + - type: nauc_mrr_at_1000_std + value: -2.101039596329006 + - type: nauc_mrr_at_100_diff1 + value: 55.622781744689384 + - type: nauc_mrr_at_100_max + value: 42.68122422079546 + - type: nauc_mrr_at_100_std + value: -2.0864750833296473 + - type: nauc_mrr_at_10_diff1 + value: 55.47063589256324 + - type: nauc_mrr_at_10_max + value: 42.653801709980115 + - type: nauc_mrr_at_10_std + value: -2.236146845988775 + - type: nauc_mrr_at_1_diff1 + value: 57.949541098815594 + - type: nauc_mrr_at_1_max + value: 40.03445493454165 + - type: nauc_mrr_at_1_std + value: -4.1963394287808296 + - type: nauc_mrr_at_20_diff1 + value: 55.56084948887483 + - type: nauc_mrr_at_20_max + value: 42.74516682641108 + - type: nauc_mrr_at_20_std + value: -2.0179679723126775 + - type: nauc_mrr_at_3_diff1 + value: 55.8149569372831 + - type: nauc_mrr_at_3_max + value: 42.521881555963255 + - type: nauc_mrr_at_3_std + value: -3.080147728695881 + - type: nauc_mrr_at_5_diff1 + value: 55.73672178151813 + - type: nauc_mrr_at_5_max + value: 42.688027056621635 + - type: nauc_mrr_at_5_std + value: -2.2754878553169107 + - type: nauc_ndcg_at_1000_diff1 + value: 54.67309684540731 + - type: nauc_ndcg_at_1000_max + value: 42.92618582060855 + - type: nauc_ndcg_at_1000_std + value: -0.7449905992343744 + - type: nauc_ndcg_at_100_diff1 + value: 54.535920001271634 + - type: nauc_ndcg_at_100_max + value: 43.06901635417981 + - type: nauc_ndcg_at_100_std + value: -0.4356522481939141 + - type: nauc_ndcg_at_10_diff1 + value: 53.7887400339396 + - type: nauc_ndcg_at_10_max + value: 42.52180603754046 + - type: nauc_ndcg_at_10_std + value: -2.0573209376366934 + - type: nauc_ndcg_at_1_diff1 + value: 57.949541098815594 + - type: nauc_ndcg_at_1_max + value: 40.03445493454165 + - type: nauc_ndcg_at_1_std + value: -4.1963394287808296 + - type: nauc_ndcg_at_20_diff1 + value: 53.99591452061374 + - type: nauc_ndcg_at_20_max + value: 43.19970652412653 + - type: nauc_ndcg_at_20_std + value: -0.42892659192992116 + - type: nauc_ndcg_at_3_diff1 + value: 54.40422384686562 + - type: nauc_ndcg_at_3_max + value: 40.9839200913677 + - type: nauc_ndcg_at_3_std + value: -4.345479256596983 + - type: nauc_ndcg_at_5_diff1 + value: 54.398899170188876 + - type: nauc_ndcg_at_5_max + value: 41.70127309489077 + - type: nauc_ndcg_at_5_std + value: -3.141973882397859 + - type: nauc_precision_at_1000_diff1 + value: -12.660446076011839 + - type: nauc_precision_at_1000_max + value: 15.594789590283098 + - type: nauc_precision_at_1000_std + value: 26.43994428225968 + - type: nauc_precision_at_100_diff1 + value: -9.952885772455668 + - type: nauc_precision_at_100_max + value: 18.684570301554498 + - type: nauc_precision_at_100_std + value: 25.759623545305576 + - type: nauc_precision_at_10_diff1 + value: 7.31811120210011 + - type: nauc_precision_at_10_max + value: 31.32986811696712 + - type: nauc_precision_at_10_std + value: 17.2400913862673 + - type: nauc_precision_at_1_diff1 + value: 57.949541098815594 + - type: nauc_precision_at_1_max + value: 40.03445493454165 + - type: nauc_precision_at_1_std + value: -4.1963394287808296 + - type: nauc_precision_at_20_diff1 + value: -0.11118712130161346 + - type: nauc_precision_at_20_max + value: 29.164830463627123 + - type: nauc_precision_at_20_std + value: 25.235641117045176 + - type: nauc_precision_at_3_diff1 + value: 31.0694415916563 + - type: nauc_precision_at_3_max + value: 40.84080602730168 + - type: nauc_precision_at_3_std + value: 4.088278071267861 + - type: nauc_precision_at_5_diff1 + value: 20.113688256987306 + - type: nauc_precision_at_5_max + value: 35.935937919944635 + - type: nauc_precision_at_5_std + value: 9.797366221144738 + - type: nauc_recall_at_1000_diff1 + value: 48.35231870903123 + - type: nauc_recall_at_1000_max + value: 72.52713791324706 + - type: nauc_recall_at_1000_std + value: 74.74636091750013 + - type: nauc_recall_at_100_diff1 + value: 46.72228412441722 + - type: nauc_recall_at_100_max + value: 57.7500229507458 + - type: nauc_recall_at_100_std + value: 35.17362598118461 + - type: nauc_recall_at_10_diff1 + value: 45.22597304556717 + - type: nauc_recall_at_10_max + value: 44.0278395267189 + - type: nauc_recall_at_10_std + value: 1.8144222907837955 + - type: nauc_recall_at_1_diff1 + value: 56.56780342832046 + - type: nauc_recall_at_1_max + value: 32.052791522050285 + - type: nauc_recall_at_1_std + value: -6.970554698390048 + - type: nauc_recall_at_20_diff1 + value: 44.318848491524555 + - type: nauc_recall_at_20_max + value: 49.816389460074745 + - type: nauc_recall_at_20_std + value: 14.458837110415553 + - type: nauc_recall_at_3_diff1 + value: 50.34057895226633 + - type: nauc_recall_at_3_max + value: 39.34951586716133 + - type: nauc_recall_at_3_std + value: -5.524466313797913 + - type: nauc_recall_at_5_diff1 + value: 49.08028080612849 + - type: nauc_recall_at_5_max + value: 41.407231348363645 + - type: nauc_recall_at_5_std + value: -1.7721775935522224 + - type: ndcg_at_1 + value: 53.480000000000004 + - type: ndcg_at_10 + value: 66.25 + - type: ndcg_at_100 + value: 69.76299999999999 + - type: ndcg_at_1000 + value: 70.308 + - type: ndcg_at_20 + value: 67.823 + - type: ndcg_at_3 + value: 60.714999999999996 + - type: ndcg_at_5 + value: 63.378 + - type: precision_at_1 + value: 53.480000000000004 + - type: precision_at_10 + value: 10.589 + - type: precision_at_100 + value: 1.32 + - type: precision_at_1000 + value: 0.13999999999999999 + - type: precision_at_20 + value: 5.787 + - type: precision_at_3 + value: 26.855 + - type: precision_at_5 + value: 18.332 + - type: recall_at_1 + value: 46.717999999999996 + - type: recall_at_10 + value: 80.207 + - type: recall_at_100 + value: 95.121 + - type: recall_at_1000 + value: 98.75500000000001 + - type: recall_at_20 + value: 86.116 + - type: recall_at_3 + value: 65.459 + - type: recall_at_5 + value: 71.985 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackGisRetrieval (default) + revision: 5003b3064772da1887988e05400cf3806fe491f2 + split: test + type: mteb/cqadupstack-gis + metrics: + - type: main_score + value: 45.883 + - type: map_at_1 + value: 30.697999999999997 + - type: map_at_10 + value: 40.327 + - type: map_at_100 + value: 41.434 + - type: map_at_1000 + value: 41.501 + - type: map_at_20 + value: 40.993 + - type: map_at_3 + value: 37.296 + - type: map_at_5 + value: 38.934999999999995 + - type: mrr_at_1 + value: 33.44632768361582 + - type: mrr_at_10 + value: 42.79459241323648 + - type: mrr_at_100 + value: 43.73677741798232 + - type: mrr_at_1000 + value: 43.781698896551056 + - type: mrr_at_20 + value: 43.397123729888584 + - type: mrr_at_3 + value: 40.09416195856873 + - type: mrr_at_5 + value: 41.613935969868145 + - type: nauc_map_at_1000_diff1 + value: 45.44421563167211 + - type: nauc_map_at_1000_max + value: 30.623506708423527 + - type: nauc_map_at_1000_std + value: -3.8101606719171643 + - type: nauc_map_at_100_diff1 + value: 45.453740774369635 + - type: nauc_map_at_100_max + value: 30.624266549501932 + - type: nauc_map_at_100_std + value: -3.7883175531232256 + - type: nauc_map_at_10_diff1 + value: 45.39658384089741 + - type: nauc_map_at_10_max + value: 30.501119139749527 + - type: nauc_map_at_10_std + value: -4.12836422418549 + - type: nauc_map_at_1_diff1 + value: 49.28905642177858 + - type: nauc_map_at_1_max + value: 28.6505285530846 + - type: nauc_map_at_1_std + value: -6.761796442459755 + - type: nauc_map_at_20_diff1 + value: 45.400552297090606 + - type: nauc_map_at_20_max + value: 30.63338514714219 + - type: nauc_map_at_20_std + value: -3.845155545827573 + - type: nauc_map_at_3_diff1 + value: 45.51557310321349 + - type: nauc_map_at_3_max + value: 28.472783894413705 + - type: nauc_map_at_3_std + value: -5.660624021294811 + - type: nauc_map_at_5_diff1 + value: 45.25576740256139 + - type: nauc_map_at_5_max + value: 29.460569720604656 + - type: nauc_map_at_5_std + value: -5.029636607852024 + - type: nauc_mrr_at_1000_diff1 + value: 44.95626167942249 + - type: nauc_mrr_at_1000_max + value: 32.1271201277396 + - type: nauc_mrr_at_1000_std + value: -1.5821651476108836 + - type: nauc_mrr_at_100_diff1 + value: 44.96101784154925 + - type: nauc_mrr_at_100_max + value: 32.1382241139764 + - type: nauc_mrr_at_100_std + value: -1.5561915559062556 + - type: nauc_mrr_at_10_diff1 + value: 44.88777443929878 + - type: nauc_mrr_at_10_max + value: 32.04353872530537 + - type: nauc_mrr_at_10_std + value: -1.7929704984616737 + - type: nauc_mrr_at_1_diff1 + value: 48.57989851004037 + - type: nauc_mrr_at_1_max + value: 31.153361810828496 + - type: nauc_mrr_at_1_std + value: -4.2236663323733765 + - type: nauc_mrr_at_20_diff1 + value: 44.918000646457706 + - type: nauc_mrr_at_20_max + value: 32.18248865789242 + - type: nauc_mrr_at_20_std + value: -1.5325561667153742 + - type: nauc_mrr_at_3_diff1 + value: 45.35365583672622 + - type: nauc_mrr_at_3_max + value: 31.102858908195845 + - type: nauc_mrr_at_3_std + value: -2.6111174959972727 + - type: nauc_mrr_at_5_diff1 + value: 44.88924139842611 + - type: nauc_mrr_at_5_max + value: 31.525528608325608 + - type: nauc_mrr_at_5_std + value: -2.2565241309789617 + - type: nauc_ndcg_at_1000_diff1 + value: 44.00702883586725 + - type: nauc_ndcg_at_1000_max + value: 32.23385705271548 + - type: nauc_ndcg_at_1000_std + value: -1.2507884164876697 + - type: nauc_ndcg_at_100_diff1 + value: 44.16248497398267 + - type: nauc_ndcg_at_100_max + value: 32.57158606632098 + - type: nauc_ndcg_at_100_std + value: -0.4549439724202447 + - type: nauc_ndcg_at_10_diff1 + value: 43.8428655070983 + - type: nauc_ndcg_at_10_max + value: 32.209130727058586 + - type: nauc_ndcg_at_10_std + value: -1.8870771688012915 + - type: nauc_ndcg_at_1_diff1 + value: 48.57989851004037 + - type: nauc_ndcg_at_1_max + value: 31.153361810828496 + - type: nauc_ndcg_at_1_std + value: -4.2236663323733765 + - type: nauc_ndcg_at_20_diff1 + value: 43.86511279185733 + - type: nauc_ndcg_at_20_max + value: 32.718221229247256 + - type: nauc_ndcg_at_20_std + value: -0.8644232305291075 + - type: nauc_ndcg_at_3_diff1 + value: 44.47446148571637 + - type: nauc_ndcg_at_3_max + value: 29.15887047666313 + - type: nauc_ndcg_at_3_std + value: -4.600240626298902 + - type: nauc_ndcg_at_5_diff1 + value: 43.81646535569093 + - type: nauc_ndcg_at_5_max + value: 30.146093839201793 + - type: nauc_ndcg_at_5_std + value: -3.640020643651651 + - type: nauc_precision_at_1000_diff1 + value: -8.959534457984326 + - type: nauc_precision_at_1000_max + value: 9.91763502436405 + - type: nauc_precision_at_1000_std + value: 12.10639950952643 + - type: nauc_precision_at_100_diff1 + value: 8.267521882809746 + - type: nauc_precision_at_100_max + value: 24.275390098170462 + - type: nauc_precision_at_100_std + value: 18.17277106012424 + - type: nauc_precision_at_10_diff1 + value: 29.821312435205595 + - type: nauc_precision_at_10_max + value: 36.18246832608729 + - type: nauc_precision_at_10_std + value: 9.141601574637006 + - type: nauc_precision_at_1_diff1 + value: 48.57989851004037 + - type: nauc_precision_at_1_max + value: 31.153361810828496 + - type: nauc_precision_at_1_std + value: -4.2236663323733765 + - type: nauc_precision_at_20_diff1 + value: 23.85062025926876 + - type: nauc_precision_at_20_max + value: 35.12799991951277 + - type: nauc_precision_at_20_std + value: 14.265595161529406 + - type: nauc_precision_at_3_diff1 + value: 38.54187441368676 + - type: nauc_precision_at_3_max + value: 31.27356112075929 + - type: nauc_precision_at_3_std + value: -0.07653265901270871 + - type: nauc_precision_at_5_diff1 + value: 34.5960935278119 + - type: nauc_precision_at_5_max + value: 32.772307659949675 + - type: nauc_precision_at_5_std + value: 2.290189139426028 + - type: nauc_recall_at_1000_diff1 + value: 18.947044948594403 + - type: nauc_recall_at_1000_max + value: 48.151023276630966 + - type: nauc_recall_at_1000_std + value: 26.693161747981197 + - type: nauc_recall_at_100_diff1 + value: 36.627729531459615 + - type: nauc_recall_at_100_max + value: 40.82537237022811 + - type: nauc_recall_at_100_std + value: 19.22310495509399 + - type: nauc_recall_at_10_diff1 + value: 37.78253026650272 + - type: nauc_recall_at_10_max + value: 35.71753676853918 + - type: nauc_recall_at_10_std + value: 3.40888122104267 + - type: nauc_recall_at_1_diff1 + value: 49.28905642177858 + - type: nauc_recall_at_1_max + value: 28.6505285530846 + - type: nauc_recall_at_1_std + value: -6.761796442459755 + - type: nauc_recall_at_20_diff1 + value: 37.083474044878805 + - type: nauc_recall_at_20_max + value: 38.49004479337876 + - type: nauc_recall_at_20_std + value: 8.756237852304002 + - type: nauc_recall_at_3_diff1 + value: 40.64710201329642 + - type: nauc_recall_at_3_max + value: 26.922789261032737 + - type: nauc_recall_at_3_std + value: -4.420858006620635 + - type: nauc_recall_at_5_diff1 + value: 38.708013355442375 + - type: nauc_recall_at_5_max + value: 29.355747723884818 + - type: nauc_recall_at_5_std + value: -2.172890467820753 + - type: ndcg_at_1 + value: 33.446 + - type: ndcg_at_10 + value: 45.883 + - type: ndcg_at_100 + value: 51.096 + - type: ndcg_at_1000 + value: 52.64 + - type: ndcg_at_20 + value: 48.156 + - type: ndcg_at_3 + value: 40.087 + - type: ndcg_at_5 + value: 42.76 + - type: precision_at_1 + value: 33.446 + - type: precision_at_10 + value: 7.0169999999999995 + - type: precision_at_100 + value: 1.011 + - type: precision_at_1000 + value: 0.11800000000000001 + - type: precision_at_20 + value: 4.051 + - type: precision_at_3 + value: 16.761 + - type: precision_at_5 + value: 11.729000000000001 + - type: recall_at_1 + value: 30.697999999999997 + - type: recall_at_10 + value: 60.702 + - type: recall_at_100 + value: 84.129 + - type: recall_at_1000 + value: 95.46 + - type: recall_at_20 + value: 69.14200000000001 + - type: recall_at_3 + value: 45.08 + - type: recall_at_5 + value: 51.376 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackMathematicaRetrieval (default) + revision: 90fceea13679c63fe563ded68f3b6f06e50061de + split: test + type: mteb/cqadupstack-mathematica + metrics: + - type: main_score + value: 37.796 + - type: map_at_1 + value: 21.426000000000002 + - type: map_at_10 + value: 31.52 + - type: map_at_100 + value: 32.927 + - type: map_at_1000 + value: 33.041 + - type: map_at_20 + value: 32.347 + - type: map_at_3 + value: 28.029 + - type: map_at_5 + value: 29.828 + - type: mrr_at_1 + value: 27.487562189054728 + - type: mrr_at_10 + value: 36.95110755745083 + - type: mrr_at_100 + value: 38.00584360466841 + - type: mrr_at_1000 + value: 38.06309113266576 + - type: mrr_at_20 + value: 37.636201994899 + - type: mrr_at_3 + value: 34.16252072968493 + - type: mrr_at_5 + value: 35.69859038142619 + - type: nauc_map_at_1000_diff1 + value: 38.39213227704596 + - type: nauc_map_at_1000_max + value: 21.57702242767287 + - type: nauc_map_at_1000_std + value: -1.0936143694367022 + - type: nauc_map_at_100_diff1 + value: 38.34996685783921 + - type: nauc_map_at_100_max + value: 21.554836186784048 + - type: nauc_map_at_100_std + value: -1.0534282832260922 + - type: nauc_map_at_10_diff1 + value: 38.521432861070345 + - type: nauc_map_at_10_max + value: 21.483601000611007 + - type: nauc_map_at_10_std + value: -1.4646817515042436 + - type: nauc_map_at_1_diff1 + value: 42.83840534981306 + - type: nauc_map_at_1_max + value: 20.6643971925017 + - type: nauc_map_at_1_std + value: -1.9814789639625436 + - type: nauc_map_at_20_diff1 + value: 38.27322603917406 + - type: nauc_map_at_20_max + value: 21.34376535841037 + - type: nauc_map_at_20_std + value: -1.3435055003496548 + - type: nauc_map_at_3_diff1 + value: 39.02735959390591 + - type: nauc_map_at_3_max + value: 20.841621815147757 + - type: nauc_map_at_3_std + value: -2.3170124589401033 + - type: nauc_map_at_5_diff1 + value: 38.66823161897012 + - type: nauc_map_at_5_max + value: 21.18719143237609 + - type: nauc_map_at_5_std + value: -1.9206249055724753 + - type: nauc_mrr_at_1000_diff1 + value: 39.25646721620463 + - type: nauc_mrr_at_1000_max + value: 24.20801953561293 + - type: nauc_mrr_at_1000_std + value: -0.07957529545135536 + - type: nauc_mrr_at_100_diff1 + value: 39.225421079182404 + - type: nauc_mrr_at_100_max + value: 24.19800686934123 + - type: nauc_mrr_at_100_std + value: -0.07066102210782718 + - type: nauc_mrr_at_10_diff1 + value: 39.328567895078166 + - type: nauc_mrr_at_10_max + value: 24.3247411149719 + - type: nauc_mrr_at_10_std + value: -0.10663356524879163 + - type: nauc_mrr_at_1_diff1 + value: 43.103676932772935 + - type: nauc_mrr_at_1_max + value: 23.107745973968402 + - type: nauc_mrr_at_1_std + value: -1.740079277634669 + - type: nauc_mrr_at_20_diff1 + value: 39.20079549946026 + - type: nauc_mrr_at_20_max + value: 24.19759412291909 + - type: nauc_mrr_at_20_std + value: -0.21846119784141474 + - type: nauc_mrr_at_3_diff1 + value: 40.003223550462344 + - type: nauc_mrr_at_3_max + value: 24.31950228354531 + - type: nauc_mrr_at_3_std + value: -0.8512369292753458 + - type: nauc_mrr_at_5_diff1 + value: 39.600956984222904 + - type: nauc_mrr_at_5_max + value: 24.222323052715392 + - type: nauc_mrr_at_5_std + value: -0.3593290219411449 + - type: nauc_ndcg_at_1000_diff1 + value: 37.44952791106159 + - type: nauc_ndcg_at_1000_max + value: 23.304806939038013 + - type: nauc_ndcg_at_1000_std + value: 1.4384351930555155 + - type: nauc_ndcg_at_100_diff1 + value: 36.58572341526991 + - type: nauc_ndcg_at_100_max + value: 23.26149202024349 + - type: nauc_ndcg_at_100_std + value: 1.8980388536069683 + - type: nauc_ndcg_at_10_diff1 + value: 37.18917185196716 + - type: nauc_ndcg_at_10_max + value: 22.96417942990795 + - type: nauc_ndcg_at_10_std + value: 0.19255110739057932 + - type: nauc_ndcg_at_1_diff1 + value: 43.103676932772935 + - type: nauc_ndcg_at_1_max + value: 23.107745973968402 + - type: nauc_ndcg_at_1_std + value: -1.740079277634669 + - type: nauc_ndcg_at_20_diff1 + value: 36.53226437322273 + - type: nauc_ndcg_at_20_max + value: 22.6090074943839 + - type: nauc_ndcg_at_20_std + value: 0.30276276893742615 + - type: nauc_ndcg_at_3_diff1 + value: 38.10825815876331 + - type: nauc_ndcg_at_3_max + value: 22.156953245617707 + - type: nauc_ndcg_at_3_std + value: -1.6981222564763174 + - type: nauc_ndcg_at_5_diff1 + value: 37.76222508240056 + - type: nauc_ndcg_at_5_max + value: 22.4760757138203 + - type: nauc_ndcg_at_5_std + value: -0.8108308533999442 + - type: nauc_precision_at_1000_diff1 + value: 3.409031463227214 + - type: nauc_precision_at_1000_max + value: 5.470385231653162 + - type: nauc_precision_at_1000_std + value: 1.2750961090108763 + - type: nauc_precision_at_100_diff1 + value: 9.164334504290373 + - type: nauc_precision_at_100_max + value: 13.59218872230791 + - type: nauc_precision_at_100_std + value: 7.013536890678054 + - type: nauc_precision_at_10_diff1 + value: 20.118158654339922 + - type: nauc_precision_at_10_max + value: 19.181237124398134 + - type: nauc_precision_at_10_std + value: 2.3515031041634398 + - type: nauc_precision_at_1_diff1 + value: 43.103676932772935 + - type: nauc_precision_at_1_max + value: 23.107745973968402 + - type: nauc_precision_at_1_std + value: -1.740079277634669 + - type: nauc_precision_at_20_diff1 + value: 15.238038636522292 + - type: nauc_precision_at_20_max + value: 15.489117413166248 + - type: nauc_precision_at_20_std + value: 2.7627367737860147 + - type: nauc_precision_at_3_diff1 + value: 32.39818661736952 + - type: nauc_precision_at_3_max + value: 22.571755119582587 + - type: nauc_precision_at_3_std + value: -0.6531438685372066 + - type: nauc_precision_at_5_diff1 + value: 26.40838498567037 + - type: nauc_precision_at_5_max + value: 20.412395112322084 + - type: nauc_precision_at_5_std + value: 0.08640629915249513 + - type: nauc_recall_at_1000_diff1 + value: 20.13879185222701 + - type: nauc_recall_at_1000_max + value: 26.222917853871465 + - type: nauc_recall_at_1000_std + value: 36.96057503157674 + - type: nauc_recall_at_100_diff1 + value: 21.433025875416217 + - type: nauc_recall_at_100_max + value: 23.164535658137694 + - type: nauc_recall_at_100_std + value: 15.692997078726567 + - type: nauc_recall_at_10_diff1 + value: 30.4863823024774 + - type: nauc_recall_at_10_max + value: 22.61787958839183 + - type: nauc_recall_at_10_std + value: 3.351042473008005 + - type: nauc_recall_at_1_diff1 + value: 42.83840534981306 + - type: nauc_recall_at_1_max + value: 20.6643971925017 + - type: nauc_recall_at_1_std + value: -1.9814789639625436 + - type: nauc_recall_at_20_diff1 + value: 26.611871709447254 + - type: nauc_recall_at_20_max + value: 21.04555203704739 + - type: nauc_recall_at_20_std + value: 3.8904519566977296 + - type: nauc_recall_at_3_diff1 + value: 33.9565557198254 + - type: nauc_recall_at_3_max + value: 21.616738927913172 + - type: nauc_recall_at_3_std + value: -0.8296072016439778 + - type: nauc_recall_at_5_diff1 + value: 33.30759305888151 + - type: nauc_recall_at_5_max + value: 22.136091554801578 + - type: nauc_recall_at_5_std + value: 0.9319140721869306 + - type: ndcg_at_1 + value: 27.488 + - type: ndcg_at_10 + value: 37.796 + - type: ndcg_at_100 + value: 43.883 + - type: ndcg_at_1000 + value: 46.215 + - type: ndcg_at_20 + value: 40.483000000000004 + - type: ndcg_at_3 + value: 31.872 + - type: ndcg_at_5 + value: 34.404 + - type: precision_at_1 + value: 27.488 + - type: precision_at_10 + value: 7.289 + - type: precision_at_100 + value: 1.17 + - type: precision_at_1000 + value: 0.148 + - type: precision_at_20 + value: 4.366 + - type: precision_at_3 + value: 15.506 + - type: precision_at_5 + value: 11.393 + - type: recall_at_1 + value: 21.426000000000002 + - type: recall_at_10 + value: 51.553000000000004 + - type: recall_at_100 + value: 77.239 + - type: recall_at_1000 + value: 93.48100000000001 + - type: recall_at_20 + value: 61.26200000000001 + - type: recall_at_3 + value: 35.102 + - type: recall_at_5 + value: 41.608000000000004 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackPhysicsRetrieval (default) + revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4 + split: test + type: mteb/cqadupstack-physics + metrics: + - type: main_score + value: 53.445 + - type: map_at_1 + value: 32.196000000000005 + - type: map_at_10 + value: 46.442 + - type: map_at_100 + value: 47.739 + - type: map_at_1000 + value: 47.837 + - type: map_at_20 + value: 47.18 + - type: map_at_3 + value: 42.455999999999996 + - type: map_at_5 + value: 44.651999999999994 + - type: mrr_at_1 + value: 40.13474494706448 + - type: mrr_at_10 + value: 52.10252532196705 + - type: mrr_at_100 + value: 52.79820072478704 + - type: mrr_at_1000 + value: 52.826261903436986 + - type: mrr_at_20 + value: 52.51913764654861 + - type: mrr_at_3 + value: 49.63105550208528 + - type: mrr_at_5 + value: 51.060314404876415 + - type: nauc_map_at_1000_diff1 + value: 51.22219263118089 + - type: nauc_map_at_1000_max + value: 38.03827817671988 + - type: nauc_map_at_1000_std + value: -9.832161001922175 + - type: nauc_map_at_100_diff1 + value: 51.193136604049386 + - type: nauc_map_at_100_max + value: 37.99935136151883 + - type: nauc_map_at_100_std + value: -9.890492633692286 + - type: nauc_map_at_10_diff1 + value: 51.21827950847157 + - type: nauc_map_at_10_max + value: 37.44792250444583 + - type: nauc_map_at_10_std + value: -10.806052338977999 + - type: nauc_map_at_1_diff1 + value: 58.178543384572876 + - type: nauc_map_at_1_max + value: 36.603116620271535 + - type: nauc_map_at_1_std + value: -11.701125296121285 + - type: nauc_map_at_20_diff1 + value: 51.17659511246892 + - type: nauc_map_at_20_max + value: 37.80552022630944 + - type: nauc_map_at_20_std + value: -10.225711470336227 + - type: nauc_map_at_3_diff1 + value: 51.78307093798811 + - type: nauc_map_at_3_max + value: 36.83735074985249 + - type: nauc_map_at_3_std + value: -10.93393134378652 + - type: nauc_map_at_5_diff1 + value: 51.22296225219469 + - type: nauc_map_at_5_max + value: 36.75437710252196 + - type: nauc_map_at_5_std + value: -11.26469528532076 + - type: nauc_mrr_at_1000_diff1 + value: 50.053153974285976 + - type: nauc_mrr_at_1000_max + value: 39.88060074773709 + - type: nauc_mrr_at_1000_std + value: -6.430263598115282 + - type: nauc_mrr_at_100_diff1 + value: 50.04492895264706 + - type: nauc_mrr_at_100_max + value: 39.89112094332879 + - type: nauc_mrr_at_100_std + value: -6.397488856079511 + - type: nauc_mrr_at_10_diff1 + value: 49.907679136667156 + - type: nauc_mrr_at_10_max + value: 39.6014368721603 + - type: nauc_mrr_at_10_std + value: -6.684492136455152 + - type: nauc_mrr_at_1_diff1 + value: 55.91787358361461 + - type: nauc_mrr_at_1_max + value: 41.990257891244234 + - type: nauc_mrr_at_1_std + value: -6.220176492762885 + - type: nauc_mrr_at_20_diff1 + value: 49.94432098122371 + - type: nauc_mrr_at_20_max + value: 39.8552975311606 + - type: nauc_mrr_at_20_std + value: -6.48720270814736 + - type: nauc_mrr_at_3_diff1 + value: 50.04108718553435 + - type: nauc_mrr_at_3_max + value: 39.537792892499006 + - type: nauc_mrr_at_3_std + value: -7.083795332696595 + - type: nauc_mrr_at_5_diff1 + value: 49.90903499103207 + - type: nauc_mrr_at_5_max + value: 39.31764526455305 + - type: nauc_mrr_at_5_std + value: -6.928346202489593 + - type: nauc_ndcg_at_1000_diff1 + value: 49.64532678615315 + - type: nauc_ndcg_at_1000_max + value: 39.244554416784396 + - type: nauc_ndcg_at_1000_std + value: -6.7235249138922395 + - type: nauc_ndcg_at_100_diff1 + value: 49.26432885639102 + - type: nauc_ndcg_at_100_max + value: 39.144367710515155 + - type: nauc_ndcg_at_100_std + value: -6.630771729553518 + - type: nauc_ndcg_at_10_diff1 + value: 48.79112716294882 + - type: nauc_ndcg_at_10_max + value: 37.252114256264434 + - type: nauc_ndcg_at_10_std + value: -10.008127831776244 + - type: nauc_ndcg_at_1_diff1 + value: 55.91787358361461 + - type: nauc_ndcg_at_1_max + value: 41.990257891244234 + - type: nauc_ndcg_at_1_std + value: -6.220176492762885 + - type: nauc_ndcg_at_20_diff1 + value: 48.69695264187268 + - type: nauc_ndcg_at_20_max + value: 38.3636363521736 + - type: nauc_ndcg_at_20_std + value: -8.50977663010693 + - type: nauc_ndcg_at_3_diff1 + value: 49.233633591644804 + - type: nauc_ndcg_at_3_max + value: 37.781556873503 + - type: nauc_ndcg_at_3_std + value: -9.460327377668014 + - type: nauc_ndcg_at_5_diff1 + value: 48.56795957964369 + - type: nauc_ndcg_at_5_max + value: 36.56295091290144 + - type: nauc_ndcg_at_5_std + value: -10.494004926806706 + - type: nauc_precision_at_1000_diff1 + value: -13.037319027940258 + - type: nauc_precision_at_1000_max + value: 3.379828097448985 + - type: nauc_precision_at_1000_std + value: 22.115873175574027 + - type: nauc_precision_at_100_diff1 + value: -9.478036292874702 + - type: nauc_precision_at_100_max + value: 10.424789570568338 + - type: nauc_precision_at_100_std + value: 21.836148384883277 + - type: nauc_precision_at_10_diff1 + value: 8.389953023130332 + - type: nauc_precision_at_10_max + value: 22.88929851663097 + - type: nauc_precision_at_10_std + value: 6.4196848727201665 + - type: nauc_precision_at_1_diff1 + value: 55.91787358361461 + - type: nauc_precision_at_1_max + value: 41.990257891244234 + - type: nauc_precision_at_1_std + value: -6.220176492762885 + - type: nauc_precision_at_20_diff1 + value: 1.5379232743752997 + - type: nauc_precision_at_20_max + value: 18.723318907854686 + - type: nauc_precision_at_20_std + value: 12.162170232837203 + - type: nauc_precision_at_3_diff1 + value: 25.970546636399465 + - type: nauc_precision_at_3_max + value: 32.72733689259369 + - type: nauc_precision_at_3_std + value: 0.38384688805260814 + - type: nauc_precision_at_5_diff1 + value: 16.37262297642082 + - type: nauc_precision_at_5_max + value: 26.266290629256407 + - type: nauc_precision_at_5_std + value: 0.751972202508073 + - type: nauc_recall_at_1000_diff1 + value: 32.969244245289396 + - type: nauc_recall_at_1000_max + value: 52.19481124070684 + - type: nauc_recall_at_1000_std + value: 37.358831762514974 + - type: nauc_recall_at_100_diff1 + value: 37.00407065257824 + - type: nauc_recall_at_100_max + value: 40.4035134408558 + - type: nauc_recall_at_100_std + value: 12.472720257388904 + - type: nauc_recall_at_10_diff1 + value: 38.48242837954449 + - type: nauc_recall_at_10_max + value: 30.24287853793808 + - type: nauc_recall_at_10_std + value: -12.044514109637145 + - type: nauc_recall_at_1_diff1 + value: 58.178543384572876 + - type: nauc_recall_at_1_max + value: 36.603116620271535 + - type: nauc_recall_at_1_std + value: -11.701125296121285 + - type: nauc_recall_at_20_diff1 + value: 36.78163521030564 + - type: nauc_recall_at_20_max + value: 34.748440544831844 + - type: nauc_recall_at_20_std + value: -6.457656404461734 + - type: nauc_recall_at_3_diff1 + value: 42.70091385736941 + - type: nauc_recall_at_3_max + value: 31.82127388838231 + - type: nauc_recall_at_3_std + value: -11.880231834282704 + - type: nauc_recall_at_5_diff1 + value: 39.927789954712566 + - type: nauc_recall_at_5_max + value: 29.464938039572075 + - type: nauc_recall_at_5_std + value: -13.24415666801013 + - type: ndcg_at_1 + value: 40.135 + - type: ndcg_at_10 + value: 53.445 + - type: ndcg_at_100 + value: 58.401 + - type: ndcg_at_1000 + value: 59.848 + - type: ndcg_at_20 + value: 55.409 + - type: ndcg_at_3 + value: 47.813 + - type: ndcg_at_5 + value: 50.488 + - type: precision_at_1 + value: 40.135 + - type: precision_at_10 + value: 10.077 + - type: precision_at_100 + value: 1.456 + - type: precision_at_1000 + value: 0.174 + - type: precision_at_20 + value: 5.756 + - type: precision_at_3 + value: 23.805 + - type: precision_at_5 + value: 16.843 + - type: recall_at_1 + value: 32.196000000000005 + - type: recall_at_10 + value: 68.217 + - type: recall_at_100 + value: 88.71900000000001 + - type: recall_at_1000 + value: 97.565 + - type: recall_at_20 + value: 74.91499999999999 + - type: recall_at_3 + value: 52.184 + - type: recall_at_5 + value: 59.289 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackProgrammersRetrieval (default) + revision: 6184bc1440d2dbc7612be22b50686b8826d22b32 + split: test + type: mteb/cqadupstack-programmers + metrics: + - type: main_score + value: 51.271 + - type: map_at_1 + value: 32.286 + - type: map_at_10 + value: 44.657000000000004 + - type: map_at_100 + value: 46.154 + - type: map_at_1000 + value: 46.245999999999995 + - type: map_at_20 + value: 45.577 + - type: map_at_3 + value: 40.646 + - type: map_at_5 + value: 43.056 + - type: mrr_at_1 + value: 39.726027397260275 + - type: mrr_at_10 + value: 50.37141226353551 + - type: mrr_at_100 + value: 51.209540975382694 + - type: mrr_at_1000 + value: 51.24334938989102 + - type: mrr_at_20 + value: 50.913600716696095 + - type: mrr_at_3 + value: 47.583713850837114 + - type: mrr_at_5 + value: 49.3131659056316 + - type: nauc_map_at_1000_diff1 + value: 40.46883153738418 + - type: nauc_map_at_1000_max + value: 29.652170875520905 + - type: nauc_map_at_1000_std + value: 0.8377261132557717 + - type: nauc_map_at_100_diff1 + value: 40.46744075731467 + - type: nauc_map_at_100_max + value: 29.67636481128142 + - type: nauc_map_at_100_std + value: 0.8884519836802501 + - type: nauc_map_at_10_diff1 + value: 40.51202569826826 + - type: nauc_map_at_10_max + value: 28.9257489655038 + - type: nauc_map_at_10_std + value: -0.15381155246840364 + - type: nauc_map_at_1_diff1 + value: 43.37074123354291 + - type: nauc_map_at_1_max + value: 24.087440506602768 + - type: nauc_map_at_1_std + value: -5.122639651397363 + - type: nauc_map_at_20_diff1 + value: 40.37188643506921 + - type: nauc_map_at_20_max + value: 29.555870496301118 + - type: nauc_map_at_20_std + value: 0.5648532024480779 + - type: nauc_map_at_3_diff1 + value: 40.716135862278186 + - type: nauc_map_at_3_max + value: 28.25137681700973 + - type: nauc_map_at_3_std + value: -1.4394527994176038 + - type: nauc_map_at_5_diff1 + value: 40.702564432900616 + - type: nauc_map_at_5_max + value: 28.353947024540897 + - type: nauc_map_at_5_std + value: -0.877368744549519 + - type: nauc_mrr_at_1000_diff1 + value: 40.71842528814642 + - type: nauc_mrr_at_1000_max + value: 32.28415412112333 + - type: nauc_mrr_at_1000_std + value: 2.37658309710263 + - type: nauc_mrr_at_100_diff1 + value: 40.727841611504566 + - type: nauc_mrr_at_100_max + value: 32.30494962796247 + - type: nauc_mrr_at_100_std + value: 2.4187023210162066 + - type: nauc_mrr_at_10_diff1 + value: 40.617542440223055 + - type: nauc_mrr_at_10_max + value: 32.03593923356149 + - type: nauc_mrr_at_10_std + value: 2.0723639712150153 + - type: nauc_mrr_at_1_diff1 + value: 43.308391651519024 + - type: nauc_mrr_at_1_max + value: 30.570125716845048 + - type: nauc_mrr_at_1_std + value: 7.100027895679755e-05 + - type: nauc_mrr_at_20_diff1 + value: 40.70114864236755 + - type: nauc_mrr_at_20_max + value: 32.26737376517803 + - type: nauc_mrr_at_20_std + value: 2.294613505333605 + - type: nauc_mrr_at_3_diff1 + value: 40.493041105259486 + - type: nauc_mrr_at_3_max + value: 32.36502952470322 + - type: nauc_mrr_at_3_std + value: 1.2416525143612114 + - type: nauc_mrr_at_5_diff1 + value: 40.48466742974365 + - type: nauc_mrr_at_5_max + value: 32.01757408839015 + - type: nauc_mrr_at_5_std + value: 1.6897434059692291 + - type: nauc_ndcg_at_1000_diff1 + value: 40.09416238253476 + - type: nauc_ndcg_at_1000_max + value: 31.43651740383964 + - type: nauc_ndcg_at_1000_std + value: 3.3000085392739615 + - type: nauc_ndcg_at_100_diff1 + value: 40.253843243910126 + - type: nauc_ndcg_at_100_max + value: 32.08661562518416 + - type: nauc_ndcg_at_100_std + value: 4.651193570771594 + - type: nauc_ndcg_at_10_diff1 + value: 39.97157348880114 + - type: nauc_ndcg_at_10_max + value: 30.15633154792771 + - type: nauc_ndcg_at_10_std + value: 1.4841091007067497 + - type: nauc_ndcg_at_1_diff1 + value: 43.308391651519024 + - type: nauc_ndcg_at_1_max + value: 30.570125716845048 + - type: nauc_ndcg_at_1_std + value: 7.100027895679755e-05 + - type: nauc_ndcg_at_20_diff1 + value: 39.75117296624936 + - type: nauc_ndcg_at_20_max + value: 31.706369825878078 + - type: nauc_ndcg_at_20_std + value: 3.175311720404324 + - type: nauc_ndcg_at_3_diff1 + value: 39.873543696643374 + - type: nauc_ndcg_at_3_max + value: 30.61510675375797 + - type: nauc_ndcg_at_3_std + value: 0.5938419988675874 + - type: nauc_ndcg_at_5_diff1 + value: 40.09469477538266 + - type: nauc_ndcg_at_5_max + value: 29.733744767668224 + - type: nauc_ndcg_at_5_std + value: 0.5446914677676432 + - type: nauc_precision_at_1000_diff1 + value: -3.4559201815461513 + - type: nauc_precision_at_1000_max + value: 4.248534523661407 + - type: nauc_precision_at_1000_std + value: 7.714623158757456 + - type: nauc_precision_at_100_diff1 + value: 0.664837507860653 + - type: nauc_precision_at_100_max + value: 13.733040712223952 + - type: nauc_precision_at_100_std + value: 17.073206692285652 + - type: nauc_precision_at_10_diff1 + value: 13.843877718215007 + - type: nauc_precision_at_10_max + value: 25.549171480505606 + - type: nauc_precision_at_10_std + value: 14.022981545864754 + - type: nauc_precision_at_1_diff1 + value: 43.308391651519024 + - type: nauc_precision_at_1_max + value: 30.570125716845048 + - type: nauc_precision_at_1_std + value: 7.100027895679755e-05 + - type: nauc_precision_at_20_diff1 + value: 7.088856747429059 + - type: nauc_precision_at_20_max + value: 23.856728008953503 + - type: nauc_precision_at_20_std + value: 17.84612403247252 + - type: nauc_precision_at_3_diff1 + value: 27.2765414441356 + - type: nauc_precision_at_3_max + value: 34.14112935222498 + - type: nauc_precision_at_3_std + value: 9.551807321595444 + - type: nauc_precision_at_5_diff1 + value: 20.264250464992838 + - type: nauc_precision_at_5_max + value: 28.871957417944856 + - type: nauc_precision_at_5_std + value: 12.059837363463684 + - type: nauc_recall_at_1000_diff1 + value: 26.986183325357803 + - type: nauc_recall_at_1000_max + value: 37.08257312546647 + - type: nauc_recall_at_1000_std + value: 40.38328159886298 + - type: nauc_recall_at_100_diff1 + value: 37.15947004173616 + - type: nauc_recall_at_100_max + value: 40.7253191145783 + - type: nauc_recall_at_100_std + value: 32.88124738114559 + - type: nauc_recall_at_10_diff1 + value: 34.883342173639434 + - type: nauc_recall_at_10_max + value: 27.442887913749715 + - type: nauc_recall_at_10_std + value: 3.970763257061315 + - type: nauc_recall_at_1_diff1 + value: 43.37074123354291 + - type: nauc_recall_at_1_max + value: 24.087440506602768 + - type: nauc_recall_at_1_std + value: -5.122639651397363 + - type: nauc_recall_at_20_diff1 + value: 33.45564577727663 + - type: nauc_recall_at_20_max + value: 33.5960673371751 + - type: nauc_recall_at_20_std + value: 11.202953326071238 + - type: nauc_recall_at_3_diff1 + value: 36.93127622587397 + - type: nauc_recall_at_3_max + value: 28.862795006648106 + - type: nauc_recall_at_3_std + value: -0.018246859072046165 + - type: nauc_recall_at_5_diff1 + value: 36.23992177849729 + - type: nauc_recall_at_5_max + value: 27.304863937695213 + - type: nauc_recall_at_5_std + value: 0.8894302819140063 + - type: ndcg_at_1 + value: 39.726 + - type: ndcg_at_10 + value: 51.271 + - type: ndcg_at_100 + value: 56.767 + - type: ndcg_at_1000 + value: 58.331 + - type: ndcg_at_20 + value: 53.813 + - type: ndcg_at_3 + value: 45.32 + - type: ndcg_at_5 + value: 48.374 + - type: precision_at_1 + value: 39.726 + - type: precision_at_10 + value: 9.532 + - type: precision_at_100 + value: 1.4200000000000002 + - type: precision_at_1000 + value: 0.172 + - type: precision_at_20 + value: 5.599 + - type: precision_at_3 + value: 21.728 + - type: precision_at_5 + value: 15.799 + - type: recall_at_1 + value: 32.286 + - type: recall_at_10 + value: 64.84400000000001 + - type: recall_at_100 + value: 87.303 + - type: recall_at_1000 + value: 97.54599999999999 + - type: recall_at_20 + value: 73.81299999999999 + - type: recall_at_3 + value: 48.497 + - type: recall_at_5 + value: 56.46600000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackStatsRetrieval (default) + revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a + split: test + type: mteb/cqadupstack-stats + metrics: + - type: main_score + value: 43.272 + - type: map_at_1 + value: 27.573999999999998 + - type: map_at_10 + value: 37.508 + - type: map_at_100 + value: 38.481 + - type: map_at_1000 + value: 38.568999999999996 + - type: map_at_20 + value: 37.974000000000004 + - type: map_at_3 + value: 34.294000000000004 + - type: map_at_5 + value: 36.019 + - type: mrr_at_1 + value: 31.28834355828221 + - type: mrr_at_10 + value: 40.46280066218714 + - type: mrr_at_100 + value: 41.21401117931536 + - type: mrr_at_1000 + value: 41.27280908559473 + - type: mrr_at_20 + value: 40.853763195604905 + - type: mrr_at_3 + value: 37.474437627811874 + - type: mrr_at_5 + value: 39.18456032719836 + - type: nauc_map_at_1000_diff1 + value: 49.06426290421037 + - type: nauc_map_at_1000_max + value: 41.01939902900332 + - type: nauc_map_at_1000_std + value: -2.200154270530929 + - type: nauc_map_at_100_diff1 + value: 49.063506225161014 + - type: nauc_map_at_100_max + value: 40.979846447451806 + - type: nauc_map_at_100_std + value: -2.2228615770946423 + - type: nauc_map_at_10_diff1 + value: 49.09924264934765 + - type: nauc_map_at_10_max + value: 40.97438053758273 + - type: nauc_map_at_10_std + value: -2.485324478806725 + - type: nauc_map_at_1_diff1 + value: 55.531554061404776 + - type: nauc_map_at_1_max + value: 40.439549779406306 + - type: nauc_map_at_1_std + value: -4.557885512257782 + - type: nauc_map_at_20_diff1 + value: 49.02757689726055 + - type: nauc_map_at_20_max + value: 40.85149923072968 + - type: nauc_map_at_20_std + value: -2.405248315780289 + - type: nauc_map_at_3_diff1 + value: 50.01952176703517 + - type: nauc_map_at_3_max + value: 40.50366969692154 + - type: nauc_map_at_3_std + value: -4.36685192128884 + - type: nauc_map_at_5_diff1 + value: 49.36953027141865 + - type: nauc_map_at_5_max + value: 41.06338727975909 + - type: nauc_map_at_5_std + value: -3.1387069399443996 + - type: nauc_mrr_at_1000_diff1 + value: 47.842810880809566 + - type: nauc_mrr_at_1000_max + value: 42.55915865845777 + - type: nauc_mrr_at_1000_std + value: 0.4107908218096477 + - type: nauc_mrr_at_100_diff1 + value: 47.84431601175813 + - type: nauc_mrr_at_100_max + value: 42.548133819833744 + - type: nauc_mrr_at_100_std + value: 0.41477319547436675 + - type: nauc_mrr_at_10_diff1 + value: 47.89192544945818 + - type: nauc_mrr_at_10_max + value: 42.741553216387466 + - type: nauc_mrr_at_10_std + value: 0.27758276370703394 + - type: nauc_mrr_at_1_diff1 + value: 53.83875008346328 + - type: nauc_mrr_at_1_max + value: 42.7138193706216 + - type: nauc_mrr_at_1_std + value: -1.2585993070573323 + - type: nauc_mrr_at_20_diff1 + value: 47.81360469913107 + - type: nauc_mrr_at_20_max + value: 42.56431869756881 + - type: nauc_mrr_at_20_std + value: 0.3971874102923386 + - type: nauc_mrr_at_3_diff1 + value: 48.48182917053402 + - type: nauc_mrr_at_3_max + value: 42.4668337512491 + - type: nauc_mrr_at_3_std + value: -1.1223757285259044 + - type: nauc_mrr_at_5_diff1 + value: 48.125055283905695 + - type: nauc_mrr_at_5_max + value: 42.76236615063886 + - type: nauc_mrr_at_5_std + value: -0.018672073587235098 + - type: nauc_ndcg_at_1000_diff1 + value: 46.27066463395065 + - type: nauc_ndcg_at_1000_max + value: 41.727638819568305 + - type: nauc_ndcg_at_1000_std + value: 1.2687860972140328 + - type: nauc_ndcg_at_100_diff1 + value: 46.22325211262054 + - type: nauc_ndcg_at_100_max + value: 41.073532324653435 + - type: nauc_ndcg_at_100_std + value: 0.9963707507261759 + - type: nauc_ndcg_at_10_diff1 + value: 46.330747152766854 + - type: nauc_ndcg_at_10_max + value: 41.149938270715495 + - type: nauc_ndcg_at_10_std + value: -0.34113384329364344 + - type: nauc_ndcg_at_1_diff1 + value: 53.83875008346328 + - type: nauc_ndcg_at_1_max + value: 42.7138193706216 + - type: nauc_ndcg_at_1_std + value: -1.2585993070573323 + - type: nauc_ndcg_at_20_diff1 + value: 46.05836075857687 + - type: nauc_ndcg_at_20_max + value: 40.6165843650717 + - type: nauc_ndcg_at_20_std + value: 0.07798286620859007 + - type: nauc_ndcg_at_3_diff1 + value: 47.7179142859349 + - type: nauc_ndcg_at_3_max + value: 41.311783353269945 + - type: nauc_ndcg_at_3_std + value: -3.0298039484986963 + - type: nauc_ndcg_at_5_diff1 + value: 46.79302008525647 + - type: nauc_ndcg_at_5_max + value: 41.558868641787065 + - type: nauc_ndcg_at_5_std + value: -1.5198103230075024 + - type: nauc_precision_at_1000_diff1 + value: -7.8947448039312915 + - type: nauc_precision_at_1000_max + value: 12.39322626936529 + - type: nauc_precision_at_1000_std + value: 14.925766815862149 + - type: nauc_precision_at_100_diff1 + value: 6.415565785526074 + - type: nauc_precision_at_100_max + value: 21.995959874137775 + - type: nauc_precision_at_100_std + value: 16.266000910963776 + - type: nauc_precision_at_10_diff1 + value: 23.126530231373376 + - type: nauc_precision_at_10_max + value: 34.44557631554908 + - type: nauc_precision_at_10_std + value: 11.692333713063038 + - type: nauc_precision_at_1_diff1 + value: 53.83875008346328 + - type: nauc_precision_at_1_max + value: 42.7138193706216 + - type: nauc_precision_at_1_std + value: -1.2585993070573323 + - type: nauc_precision_at_20_diff1 + value: 18.230473575881636 + - type: nauc_precision_at_20_max + value: 29.96947955026763 + - type: nauc_precision_at_20_std + value: 13.257695208656104 + - type: nauc_precision_at_3_diff1 + value: 36.868460910956735 + - type: nauc_precision_at_3_max + value: 39.40948041507801 + - type: nauc_precision_at_3_std + value: 1.7269999007621786 + - type: nauc_precision_at_5_diff1 + value: 31.539029398660674 + - type: nauc_precision_at_5_max + value: 40.18928818387894 + - type: nauc_precision_at_5_std + value: 7.533304408361132 + - type: nauc_recall_at_1000_diff1 + value: 12.781679703452658 + - type: nauc_recall_at_1000_max + value: 44.85343253905329 + - type: nauc_recall_at_1000_std + value: 43.114714361093 + - type: nauc_recall_at_100_diff1 + value: 30.927571640069534 + - type: nauc_recall_at_100_max + value: 34.70935438104178 + - type: nauc_recall_at_100_std + value: 14.536465474667759 + - type: nauc_recall_at_10_diff1 + value: 36.93692573380035 + - type: nauc_recall_at_10_max + value: 37.5804099530568 + - type: nauc_recall_at_10_std + value: 3.6357734847934067 + - type: nauc_recall_at_1_diff1 + value: 55.531554061404776 + - type: nauc_recall_at_1_max + value: 40.439549779406306 + - type: nauc_recall_at_1_std + value: -4.557885512257782 + - type: nauc_recall_at_20_diff1 + value: 34.19908572292086 + - type: nauc_recall_at_20_max + value: 34.29681260598194 + - type: nauc_recall_at_20_std + value: 5.28270418723725 + - type: nauc_recall_at_3_diff1 + value: 42.61233762231971 + - type: nauc_recall_at_3_max + value: 38.60052516717657 + - type: nauc_recall_at_3_std + value: -4.0375769241977535 + - type: nauc_recall_at_5_diff1 + value: 40.023153410242216 + - type: nauc_recall_at_5_max + value: 39.303505654316275 + - type: nauc_recall_at_5_std + value: 0.15791499220308775 + - type: ndcg_at_1 + value: 31.288 + - type: ndcg_at_10 + value: 43.272 + - type: ndcg_at_100 + value: 47.845 + - type: ndcg_at_1000 + value: 49.82 + - type: ndcg_at_20 + value: 44.766 + - type: ndcg_at_3 + value: 37.362 + - type: ndcg_at_5 + value: 40.119 + - type: precision_at_1 + value: 31.288 + - type: precision_at_10 + value: 7.163 + - type: precision_at_100 + value: 1.028 + - type: precision_at_1000 + value: 0.127 + - type: precision_at_20 + value: 3.972 + - type: precision_at_3 + value: 16.512999999999998 + - type: precision_at_5 + value: 11.687 + - type: recall_at_1 + value: 27.573999999999998 + - type: recall_at_10 + value: 57.955 + - type: recall_at_100 + value: 78.597 + - type: recall_at_1000 + value: 92.666 + - type: recall_at_20 + value: 63.529 + - type: recall_at_3 + value: 41.696 + - type: recall_at_5 + value: 48.557 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackTexRetrieval (default) + revision: 46989137a86843e03a6195de44b09deda022eec7 + split: test + type: mteb/cqadupstack-tex + metrics: + - type: main_score + value: 36.359 + - type: map_at_1 + value: 21.185000000000002 + - type: map_at_10 + value: 30.641000000000002 + - type: map_at_100 + value: 31.898 + - type: map_at_1000 + value: 32.019 + - type: map_at_20 + value: 31.316 + - type: map_at_3 + value: 27.679 + - type: map_at_5 + value: 29.296 + - type: mrr_at_1 + value: 26.25602202339986 + - type: mrr_at_10 + value: 35.29312642698747 + - type: mrr_at_100 + value: 36.188407203218766 + - type: mrr_at_1000 + value: 36.25754301361426 + - type: mrr_at_20 + value: 35.79539901778623 + - type: mrr_at_3 + value: 32.81715989905948 + - type: mrr_at_5 + value: 34.17297545308569 + - type: nauc_map_at_1000_diff1 + value: 38.16046480703112 + - type: nauc_map_at_1000_max + value: 30.36053187595561 + - type: nauc_map_at_1000_std + value: 1.6902772643396222 + - type: nauc_map_at_100_diff1 + value: 38.12738352750608 + - type: nauc_map_at_100_max + value: 30.329169999774646 + - type: nauc_map_at_100_std + value: 1.6725307136499883 + - type: nauc_map_at_10_diff1 + value: 38.16743409180553 + - type: nauc_map_at_10_max + value: 30.15394602944772 + - type: nauc_map_at_10_std + value: 1.0671899606243178 + - type: nauc_map_at_1_diff1 + value: 42.88589281960491 + - type: nauc_map_at_1_max + value: 27.638755108982636 + - type: nauc_map_at_1_std + value: -2.1585002712244754 + - type: nauc_map_at_20_diff1 + value: 38.09582706751841 + - type: nauc_map_at_20_max + value: 30.27946595751881 + - type: nauc_map_at_20_std + value: 1.3617258672099988 + - type: nauc_map_at_3_diff1 + value: 38.733121673764536 + - type: nauc_map_at_3_max + value: 29.395446178182393 + - type: nauc_map_at_3_std + value: -0.0658653746179121 + - type: nauc_map_at_5_diff1 + value: 38.50432551367878 + - type: nauc_map_at_5_max + value: 30.036554522323627 + - type: nauc_map_at_5_std + value: 0.416230033217505 + - type: nauc_mrr_at_1000_diff1 + value: 38.425309352314635 + - type: nauc_mrr_at_1000_max + value: 32.249621875336544 + - type: nauc_mrr_at_1000_std + value: 3.6699794559835595 + - type: nauc_mrr_at_100_diff1 + value: 38.40802118853337 + - type: nauc_mrr_at_100_max + value: 32.250667773123894 + - type: nauc_mrr_at_100_std + value: 3.689484045596767 + - type: nauc_mrr_at_10_diff1 + value: 38.434828021383716 + - type: nauc_mrr_at_10_max + value: 32.23530930362712 + - type: nauc_mrr_at_10_std + value: 3.344606244404385 + - type: nauc_mrr_at_1_diff1 + value: 42.27996089725328 + - type: nauc_mrr_at_1_max + value: 31.1006789115586 + - type: nauc_mrr_at_1_std + value: 1.646987215555319 + - type: nauc_mrr_at_20_diff1 + value: 38.39605979789424 + - type: nauc_mrr_at_20_max + value: 32.252777969161826 + - type: nauc_mrr_at_20_std + value: 3.5817305258254653 + - type: nauc_mrr_at_3_diff1 + value: 38.77676635162028 + - type: nauc_mrr_at_3_max + value: 32.235532396942446 + - type: nauc_mrr_at_3_std + value: 2.749558327534701 + - type: nauc_mrr_at_5_diff1 + value: 38.79371056915482 + - type: nauc_mrr_at_5_max + value: 32.290100031888464 + - type: nauc_mrr_at_5_std + value: 3.0340477396473444 + - type: nauc_ndcg_at_1000_diff1 + value: 37.01652405570216 + - type: nauc_ndcg_at_1000_max + value: 31.608748461137388 + - type: nauc_ndcg_at_1000_std + value: 5.0352869932226225 + - type: nauc_ndcg_at_100_diff1 + value: 36.461139070751074 + - type: nauc_ndcg_at_100_max + value: 31.39439835525597 + - type: nauc_ndcg_at_100_std + value: 5.445572526202663 + - type: nauc_ndcg_at_10_diff1 + value: 36.85387632539878 + - type: nauc_ndcg_at_10_max + value: 31.26234336791873 + - type: nauc_ndcg_at_10_std + value: 3.1005021081215176 + - type: nauc_ndcg_at_1_diff1 + value: 42.27996089725328 + - type: nauc_ndcg_at_1_max + value: 31.1006789115586 + - type: nauc_ndcg_at_1_std + value: 1.646987215555319 + - type: nauc_ndcg_at_20_diff1 + value: 36.51553821000284 + - type: nauc_ndcg_at_20_max + value: 31.418965121851656 + - type: nauc_ndcg_at_20_std + value: 3.993702058497816 + - type: nauc_ndcg_at_3_diff1 + value: 37.97395959994847 + - type: nauc_ndcg_at_3_max + value: 30.849714565509522 + - type: nauc_ndcg_at_3_std + value: 1.5592033678158221 + - type: nauc_ndcg_at_5_diff1 + value: 37.757846175886755 + - type: nauc_ndcg_at_5_max + value: 31.370410033289236 + - type: nauc_ndcg_at_5_std + value: 2.1125862421735295 + - type: nauc_precision_at_1000_diff1 + value: 5.410576728143286 + - type: nauc_precision_at_1000_max + value: 16.450242683836517 + - type: nauc_precision_at_1000_std + value: 14.678651006175524 + - type: nauc_precision_at_100_diff1 + value: 11.162025150042357 + - type: nauc_precision_at_100_max + value: 21.186053090176053 + - type: nauc_precision_at_100_std + value: 18.818230269868486 + - type: nauc_precision_at_10_diff1 + value: 23.61266483996461 + - type: nauc_precision_at_10_max + value: 30.15584300411177 + - type: nauc_precision_at_10_std + value: 10.907073711430778 + - type: nauc_precision_at_1_diff1 + value: 42.27996089725328 + - type: nauc_precision_at_1_max + value: 31.1006789115586 + - type: nauc_precision_at_1_std + value: 1.646987215555319 + - type: nauc_precision_at_20_diff1 + value: 19.280674004990168 + - type: nauc_precision_at_20_max + value: 27.867307026052814 + - type: nauc_precision_at_20_std + value: 13.729376132169113 + - type: nauc_precision_at_3_diff1 + value: 32.12183061531225 + - type: nauc_precision_at_3_max + value: 32.94477748849579 + - type: nauc_precision_at_3_std + value: 5.743302661255147 + - type: nauc_precision_at_5_diff1 + value: 28.89911521350784 + - type: nauc_precision_at_5_max + value: 32.81821744704191 + - type: nauc_precision_at_5_std + value: 7.596034152398431 + - type: nauc_recall_at_1000_diff1 + value: 27.49345184387517 + - type: nauc_recall_at_1000_max + value: 33.674225830990245 + - type: nauc_recall_at_1000_std + value: 28.745953974124106 + - type: nauc_recall_at_100_diff1 + value: 25.924079484606082 + - type: nauc_recall_at_100_max + value: 29.490098172371905 + - type: nauc_recall_at_100_std + value: 19.850728016506043 + - type: nauc_recall_at_10_diff1 + value: 29.933074832653922 + - type: nauc_recall_at_10_max + value: 29.23627074711927 + - type: nauc_recall_at_10_std + value: 5.931394949361693 + - type: nauc_recall_at_1_diff1 + value: 42.88589281960491 + - type: nauc_recall_at_1_max + value: 27.638755108982636 + - type: nauc_recall_at_1_std + value: -2.1585002712244754 + - type: nauc_recall_at_20_diff1 + value: 28.097884739603625 + - type: nauc_recall_at_20_max + value: 29.624267934012323 + - type: nauc_recall_at_20_std + value: 9.338407690108848 + - type: nauc_recall_at_3_diff1 + value: 34.29381708797508 + - type: nauc_recall_at_3_max + value: 29.085559872389737 + - type: nauc_recall_at_3_std + value: 1.8479356944857943 + - type: nauc_recall_at_5_diff1 + value: 33.278226432605834 + - type: nauc_recall_at_5_max + value: 29.834238279294205 + - type: nauc_recall_at_5_std + value: 3.1314845704874315 + - type: ndcg_at_1 + value: 26.256 + - type: ndcg_at_10 + value: 36.359 + - type: ndcg_at_100 + value: 41.847 + - type: ndcg_at_1000 + value: 44.330999999999996 + - type: ndcg_at_20 + value: 38.428000000000004 + - type: ndcg_at_3 + value: 31.334 + - type: ndcg_at_5 + value: 33.592 + - type: precision_at_1 + value: 26.256 + - type: precision_at_10 + value: 6.723999999999999 + - type: precision_at_100 + value: 1.11 + - type: precision_at_1000 + value: 0.15 + - type: precision_at_20 + value: 4.006 + - type: precision_at_3 + value: 14.946000000000002 + - type: precision_at_5 + value: 10.812 + - type: recall_at_1 + value: 21.185000000000002 + - type: recall_at_10 + value: 48.948 + - type: recall_at_100 + value: 73.089 + - type: recall_at_1000 + value: 90.298 + - type: recall_at_20 + value: 56.505 + - type: recall_at_3 + value: 34.693000000000005 + - type: recall_at_5 + value: 40.602 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackUnixRetrieval (default) + revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53 + split: test + type: mteb/cqadupstack-unix + metrics: + - type: main_score + value: 49.946 + - type: map_at_1 + value: 32.638 + - type: map_at_10 + value: 43.723 + - type: map_at_100 + value: 45.029 + - type: map_at_1000 + value: 45.109 + - type: map_at_20 + value: 44.501000000000005 + - type: map_at_3 + value: 40.103 + - type: map_at_5 + value: 42.083 + - type: mrr_at_1 + value: 38.899253731343286 + - type: mrr_at_10 + value: 48.39563343994311 + - type: mrr_at_100 + value: 49.2469884372283 + - type: mrr_at_1000 + value: 49.28304669740568 + - type: mrr_at_20 + value: 48.91859703686785 + - type: mrr_at_3 + value: 45.50684079601986 + - type: mrr_at_5 + value: 47.13463930348251 + - type: nauc_map_at_1000_diff1 + value: 50.8791568455423 + - type: nauc_map_at_1000_max + value: 40.78958778654321 + - type: nauc_map_at_1000_std + value: -4.839368962096667 + - type: nauc_map_at_100_diff1 + value: 50.857380318701765 + - type: nauc_map_at_100_max + value: 40.78538838843207 + - type: nauc_map_at_100_std + value: -4.847773242222035 + - type: nauc_map_at_10_diff1 + value: 50.75900664858353 + - type: nauc_map_at_10_max + value: 40.63786949124477 + - type: nauc_map_at_10_std + value: -5.106628865250861 + - type: nauc_map_at_1_diff1 + value: 56.72587160332597 + - type: nauc_map_at_1_max + value: 38.883508123710456 + - type: nauc_map_at_1_std + value: -7.118983929025435 + - type: nauc_map_at_20_diff1 + value: 50.796716227156104 + - type: nauc_map_at_20_max + value: 40.65011016876902 + - type: nauc_map_at_20_std + value: -4.892111467515356 + - type: nauc_map_at_3_diff1 + value: 52.00124582999345 + - type: nauc_map_at_3_max + value: 39.81668031503991 + - type: nauc_map_at_3_std + value: -6.22819599407076 + - type: nauc_map_at_5_diff1 + value: 50.88403186997827 + - type: nauc_map_at_5_max + value: 40.3056617462873 + - type: nauc_map_at_5_std + value: -5.514293793087481 + - type: nauc_mrr_at_1000_diff1 + value: 50.44454724490139 + - type: nauc_mrr_at_1000_max + value: 41.93698845640627 + - type: nauc_mrr_at_1000_std + value: -4.346444363238604 + - type: nauc_mrr_at_100_diff1 + value: 50.415801928259505 + - type: nauc_mrr_at_100_max + value: 41.91742893064303 + - type: nauc_mrr_at_100_std + value: -4.3596953645526115 + - type: nauc_mrr_at_10_diff1 + value: 50.27723830728065 + - type: nauc_mrr_at_10_max + value: 41.94210114330855 + - type: nauc_mrr_at_10_std + value: -4.522709614858563 + - type: nauc_mrr_at_1_diff1 + value: 56.27211140297573 + - type: nauc_mrr_at_1_max + value: 42.862979332085295 + - type: nauc_mrr_at_1_std + value: -5.436251223101583 + - type: nauc_mrr_at_20_diff1 + value: 50.349888540080244 + - type: nauc_mrr_at_20_max + value: 41.85347369204116 + - type: nauc_mrr_at_20_std + value: -4.354685028176137 + - type: nauc_mrr_at_3_diff1 + value: 51.26970415464677 + - type: nauc_mrr_at_3_max + value: 42.10814754779768 + - type: nauc_mrr_at_3_std + value: -4.985601024186455 + - type: nauc_mrr_at_5_diff1 + value: 50.41484164474821 + - type: nauc_mrr_at_5_max + value: 42.18066893866459 + - type: nauc_mrr_at_5_std + value: -4.482892639863031 + - type: nauc_ndcg_at_1000_diff1 + value: 49.42610425116412 + - type: nauc_ndcg_at_1000_max + value: 41.5045850587587 + - type: nauc_ndcg_at_1000_std + value: -3.10124616756314 + - type: nauc_ndcg_at_100_diff1 + value: 48.83053106960371 + - type: nauc_ndcg_at_100_max + value: 41.28017463700636 + - type: nauc_ndcg_at_100_std + value: -2.969068144903341 + - type: nauc_ndcg_at_10_diff1 + value: 48.20480088983651 + - type: nauc_ndcg_at_10_max + value: 40.82637884697993 + - type: nauc_ndcg_at_10_std + value: -4.212585519019192 + - type: nauc_ndcg_at_1_diff1 + value: 56.27211140297573 + - type: nauc_ndcg_at_1_max + value: 42.862979332085295 + - type: nauc_ndcg_at_1_std + value: -5.436251223101583 + - type: nauc_ndcg_at_20_diff1 + value: 48.46912806126355 + - type: nauc_ndcg_at_20_max + value: 40.72782242537804 + - type: nauc_ndcg_at_20_std + value: -3.2719477151187495 + - type: nauc_ndcg_at_3_diff1 + value: 50.1358795807417 + - type: nauc_ndcg_at_3_max + value: 40.50257341445765 + - type: nauc_ndcg_at_3_std + value: -5.454654002201218 + - type: nauc_ndcg_at_5_diff1 + value: 48.33079341396385 + - type: nauc_ndcg_at_5_max + value: 40.42948325530862 + - type: nauc_ndcg_at_5_std + value: -4.822844888589279 + - type: nauc_precision_at_1000_diff1 + value: -13.918887071131945 + - type: nauc_precision_at_1000_max + value: -2.525574289407267 + - type: nauc_precision_at_1000_std + value: 1.9950185975694454 + - type: nauc_precision_at_100_diff1 + value: -4.674405832799318 + - type: nauc_precision_at_100_max + value: 9.86848884486232 + - type: nauc_precision_at_100_std + value: 7.44775275660397 + - type: nauc_precision_at_10_diff1 + value: 15.848995829385624 + - type: nauc_precision_at_10_max + value: 28.04686574011267 + - type: nauc_precision_at_10_std + value: 3.181341100700103 + - type: nauc_precision_at_1_diff1 + value: 56.27211140297573 + - type: nauc_precision_at_1_max + value: 42.862979332085295 + - type: nauc_precision_at_1_std + value: -5.436251223101583 + - type: nauc_precision_at_20_diff1 + value: 8.561566624881575 + - type: nauc_precision_at_20_max + value: 21.627703407635245 + - type: nauc_precision_at_20_std + value: 5.760406412600719 + - type: nauc_precision_at_3_diff1 + value: 34.671626732535636 + - type: nauc_precision_at_3_max + value: 38.33754394515895 + - type: nauc_precision_at_3_std + value: -1.1225302010346443 + - type: nauc_precision_at_5_diff1 + value: 25.079459001839126 + - type: nauc_precision_at_5_max + value: 34.84063738508325 + - type: nauc_precision_at_5_std + value: 0.9453442491539544 + - type: nauc_recall_at_1000_diff1 + value: 41.03367721970914 + - type: nauc_recall_at_1000_max + value: 56.449777595386784 + - type: nauc_recall_at_1000_std + value: 44.514154744595004 + - type: nauc_recall_at_100_diff1 + value: 34.052938367936655 + - type: nauc_recall_at_100_max + value: 38.15032421480224 + - type: nauc_recall_at_100_std + value: 10.571479689335082 + - type: nauc_recall_at_10_diff1 + value: 37.30360739621019 + - type: nauc_recall_at_10_max + value: 36.84373887888991 + - type: nauc_recall_at_10_std + value: -2.1642057178707477 + - type: nauc_recall_at_1_diff1 + value: 56.72587160332597 + - type: nauc_recall_at_1_max + value: 38.883508123710456 + - type: nauc_recall_at_1_std + value: -7.118983929025435 + - type: nauc_recall_at_20_diff1 + value: 36.45521763229933 + - type: nauc_recall_at_20_max + value: 34.92427670207214 + - type: nauc_recall_at_20_std + value: 2.61715702817496 + - type: nauc_recall_at_3_diff1 + value: 45.0737410667599 + - type: nauc_recall_at_3_max + value: 37.448388262718794 + - type: nauc_recall_at_3_std + value: -5.4453210659266045 + - type: nauc_recall_at_5_diff1 + value: 39.2612920289044 + - type: nauc_recall_at_5_max + value: 37.07859910096755 + - type: nauc_recall_at_5_std + value: -2.9224584194325307 + - type: ndcg_at_1 + value: 38.899 + - type: ndcg_at_10 + value: 49.946 + - type: ndcg_at_100 + value: 55.33 + - type: ndcg_at_1000 + value: 56.854000000000006 + - type: ndcg_at_20 + value: 52.261 + - type: ndcg_at_3 + value: 43.912 + - type: ndcg_at_5 + value: 46.707 + - type: precision_at_1 + value: 38.899 + - type: precision_at_10 + value: 8.657 + - type: precision_at_100 + value: 1.256 + - type: precision_at_1000 + value: 0.148 + - type: precision_at_20 + value: 5.0 + - type: precision_at_3 + value: 20.18 + - type: precision_at_5 + value: 14.235000000000001 + - type: recall_at_1 + value: 32.638 + - type: recall_at_10 + value: 64.012 + - type: recall_at_100 + value: 86.895 + - type: recall_at_1000 + value: 96.965 + - type: recall_at_20 + value: 72.312 + - type: recall_at_3 + value: 47.388999999999996 + - type: recall_at_5 + value: 54.795 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackWebmastersRetrieval (default) + revision: 160c094312a0e1facb97e55eeddb698c0abe3571 + split: test + type: mteb/cqadupstack-webmasters + metrics: + - type: main_score + value: 48.339999999999996 + - type: map_at_1 + value: 30.858999999999998 + - type: map_at_10 + value: 41.533 + - type: map_at_100 + value: 43.252 + - type: map_at_1000 + value: 43.514 + - type: map_at_20 + value: 42.402 + - type: map_at_3 + value: 37.876 + - type: map_at_5 + value: 39.646 + - type: mrr_at_1 + value: 36.95652173913043 + - type: mrr_at_10 + value: 46.25078423991467 + - type: mrr_at_100 + value: 47.10771896017969 + - type: mrr_at_1000 + value: 47.15686026835414 + - type: mrr_at_20 + value: 46.794296990089876 + - type: mrr_at_3 + value: 43.24769433465087 + - type: mrr_at_5 + value: 44.66073781291173 + - type: nauc_map_at_1000_diff1 + value: 44.8742462734827 + - type: nauc_map_at_1000_max + value: 34.26660109331205 + - type: nauc_map_at_1000_std + value: 0.7879225359199136 + - type: nauc_map_at_100_diff1 + value: 45.11128919626894 + - type: nauc_map_at_100_max + value: 34.445153578600184 + - type: nauc_map_at_100_std + value: 0.4506104132623201 + - type: nauc_map_at_10_diff1 + value: 45.15551545181208 + - type: nauc_map_at_10_max + value: 34.13473008892501 + - type: nauc_map_at_10_std + value: -0.783913814990331 + - type: nauc_map_at_1_diff1 + value: 49.95390841422152 + - type: nauc_map_at_1_max + value: 31.589951345058694 + - type: nauc_map_at_1_std + value: -1.2044168463130798 + - type: nauc_map_at_20_diff1 + value: 45.292927556411755 + - type: nauc_map_at_20_max + value: 34.356211052351426 + - type: nauc_map_at_20_std + value: -0.17967800749896154 + - type: nauc_map_at_3_diff1 + value: 45.96161928859822 + - type: nauc_map_at_3_max + value: 33.00731933082887 + - type: nauc_map_at_3_std + value: -2.7098696875246837 + - type: nauc_map_at_5_diff1 + value: 46.070994971335665 + - type: nauc_map_at_5_max + value: 33.777381963682814 + - type: nauc_map_at_5_std + value: -2.6620711632122456 + - type: nauc_mrr_at_1000_diff1 + value: 42.567803205401994 + - type: nauc_mrr_at_1000_max + value: 34.16209638667259 + - type: nauc_mrr_at_1000_std + value: 1.854371152264505 + - type: nauc_mrr_at_100_diff1 + value: 42.58233265042619 + - type: nauc_mrr_at_100_max + value: 34.16078402900575 + - type: nauc_mrr_at_100_std + value: 1.8898694853453202 + - type: nauc_mrr_at_10_diff1 + value: 42.339802855902754 + - type: nauc_mrr_at_10_max + value: 34.09489835605403 + - type: nauc_mrr_at_10_std + value: 1.5465761220587855 + - type: nauc_mrr_at_1_diff1 + value: 44.694834982501746 + - type: nauc_mrr_at_1_max + value: 33.33171634199177 + - type: nauc_mrr_at_1_std + value: 0.6606279730955905 + - type: nauc_mrr_at_20_diff1 + value: 42.51728546308409 + - type: nauc_mrr_at_20_max + value: 34.036696294772305 + - type: nauc_mrr_at_20_std + value: 1.8755809344413583 + - type: nauc_mrr_at_3_diff1 + value: 42.40741534403617 + - type: nauc_mrr_at_3_max + value: 34.56073599089024 + - type: nauc_mrr_at_3_std + value: 0.14012805364017056 + - type: nauc_mrr_at_5_diff1 + value: 42.8124767186626 + - type: nauc_mrr_at_5_max + value: 34.53104798912329 + - type: nauc_mrr_at_5_std + value: 0.6346454338636053 + - type: nauc_ndcg_at_1000_diff1 + value: 43.241626646666674 + - type: nauc_ndcg_at_1000_max + value: 34.65608247753032 + - type: nauc_ndcg_at_1000_std + value: 3.760305660330352 + - type: nauc_ndcg_at_100_diff1 + value: 43.91911696270587 + - type: nauc_ndcg_at_100_max + value: 35.054324362213386 + - type: nauc_ndcg_at_100_std + value: 4.7041380350719955 + - type: nauc_ndcg_at_10_diff1 + value: 42.287646669915176 + - type: nauc_ndcg_at_10_max + value: 34.24489242953701 + - type: nauc_ndcg_at_10_std + value: 2.3378314595461074 + - type: nauc_ndcg_at_1_diff1 + value: 44.694834982501746 + - type: nauc_ndcg_at_1_max + value: 33.33171634199177 + - type: nauc_ndcg_at_1_std + value: 0.6606279730955905 + - type: nauc_ndcg_at_20_diff1 + value: 43.292614079622346 + - type: nauc_ndcg_at_20_max + value: 34.24974830610893 + - type: nauc_ndcg_at_20_std + value: 3.52776120828904 + - type: nauc_ndcg_at_3_diff1 + value: 42.56835898759269 + - type: nauc_ndcg_at_3_max + value: 34.312309582717695 + - type: nauc_ndcg_at_3_std + value: -0.4203822524036711 + - type: nauc_ndcg_at_5_diff1 + value: 43.70979311625624 + - type: nauc_ndcg_at_5_max + value: 34.63486843737983 + - type: nauc_ndcg_at_5_std + value: -0.3630888557962404 + - type: nauc_precision_at_1000_diff1 + value: -23.70803212249293 + - type: nauc_precision_at_1000_max + value: -12.248365214738262 + - type: nauc_precision_at_1000_std + value: 28.266372479185275 + - type: nauc_precision_at_100_diff1 + value: -14.928790999910607 + - type: nauc_precision_at_100_max + value: -0.26814822044508746 + - type: nauc_precision_at_100_std + value: 25.090899959288386 + - type: nauc_precision_at_10_diff1 + value: 3.777220446004569 + - type: nauc_precision_at_10_max + value: 22.773277668428992 + - type: nauc_precision_at_10_std + value: 17.389305549274575 + - type: nauc_precision_at_1_diff1 + value: 44.694834982501746 + - type: nauc_precision_at_1_max + value: 33.33171634199177 + - type: nauc_precision_at_1_std + value: 0.6606279730955905 + - type: nauc_precision_at_20_diff1 + value: -3.5124634687659877 + - type: nauc_precision_at_20_max + value: 14.63988721915741 + - type: nauc_precision_at_20_std + value: 21.8509732430562 + - type: nauc_precision_at_3_diff1 + value: 24.037612546834257 + - type: nauc_precision_at_3_max + value: 30.54263938738208 + - type: nauc_precision_at_3_std + value: 2.9977475200575165 + - type: nauc_precision_at_5_diff1 + value: 17.858088916009176 + - type: nauc_precision_at_5_max + value: 28.645521709350007 + - type: nauc_precision_at_5_std + value: 6.839474761801788 + - type: nauc_recall_at_1000_diff1 + value: 17.9753435278599 + - type: nauc_recall_at_1000_max + value: 37.90563966690478 + - type: nauc_recall_at_1000_std + value: 58.03220658909679 + - type: nauc_recall_at_100_diff1 + value: 41.4944360952115 + - type: nauc_recall_at_100_max + value: 37.087133604547326 + - type: nauc_recall_at_100_std + value: 31.36793074112707 + - type: nauc_recall_at_10_diff1 + value: 35.606768311101405 + - type: nauc_recall_at_10_max + value: 30.169326859416145 + - type: nauc_recall_at_10_std + value: 5.365538893341898 + - type: nauc_recall_at_1_diff1 + value: 49.95390841422152 + - type: nauc_recall_at_1_max + value: 31.589951345058694 + - type: nauc_recall_at_1_std + value: -1.2044168463130798 + - type: nauc_recall_at_20_diff1 + value: 37.60927011058002 + - type: nauc_recall_at_20_max + value: 29.568944636689164 + - type: nauc_recall_at_20_std + value: 12.926659896720466 + - type: nauc_recall_at_3_diff1 + value: 41.863264250476576 + - type: nauc_recall_at_3_max + value: 32.385221184845406 + - type: nauc_recall_at_3_std + value: -3.7474066203696235 + - type: nauc_recall_at_5_diff1 + value: 42.101661049235986 + - type: nauc_recall_at_5_max + value: 32.567627924051564 + - type: nauc_recall_at_5_std + value: -3.8697693428570012 + - type: ndcg_at_1 + value: 36.957 + - type: ndcg_at_10 + value: 48.339999999999996 + - type: ndcg_at_100 + value: 53.763000000000005 + - type: ndcg_at_1000 + value: 55.849000000000004 + - type: ndcg_at_20 + value: 50.517999999999994 + - type: ndcg_at_3 + value: 42.308 + - type: ndcg_at_5 + value: 44.527 + - type: precision_at_1 + value: 36.957 + - type: precision_at_10 + value: 9.347999999999999 + - type: precision_at_100 + value: 1.7870000000000001 + - type: precision_at_1000 + value: 0.259 + - type: precision_at_20 + value: 5.791 + - type: precision_at_3 + value: 19.631 + - type: precision_at_5 + value: 14.308000000000002 + - type: recall_at_1 + value: 30.858999999999998 + - type: recall_at_10 + value: 61.873 + - type: recall_at_100 + value: 85.229 + - type: recall_at_1000 + value: 97.844 + - type: recall_at_20 + value: 70.226 + - type: recall_at_3 + value: 44.269 + - type: recall_at_5 + value: 50.019000000000005 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackWordpressRetrieval (default) + revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4 + split: test + type: mteb/cqadupstack-wordpress + metrics: + - type: main_score + value: 40.929 + - type: map_at_1 + value: 26.3 + - type: map_at_10 + value: 35.693999999999996 + - type: map_at_100 + value: 36.732 + - type: map_at_1000 + value: 36.836999999999996 + - type: map_at_20 + value: 36.282 + - type: map_at_3 + value: 33.196999999999996 + - type: map_at_5 + value: 34.549 + - type: mrr_at_1 + value: 29.02033271719039 + - type: mrr_at_10 + value: 38.31147492885017 + - type: mrr_at_100 + value: 39.21471696616997 + - type: mrr_at_1000 + value: 39.278281346737195 + - type: mrr_at_20 + value: 38.82211296184961 + - type: mrr_at_3 + value: 36.1059765865681 + - type: mrr_at_5 + value: 37.29821318545904 + - type: nauc_map_at_1000_diff1 + value: 44.49970836853861 + - type: nauc_map_at_1000_max + value: 29.77152600812978 + - type: nauc_map_at_1000_std + value: 2.371611116941705 + - type: nauc_map_at_100_diff1 + value: 44.50435385542096 + - type: nauc_map_at_100_max + value: 29.783880224370968 + - type: nauc_map_at_100_std + value: 2.3901789050901416 + - type: nauc_map_at_10_diff1 + value: 44.409408206823926 + - type: nauc_map_at_10_max + value: 29.716512955118407 + - type: nauc_map_at_10_std + value: 1.9960391417306471 + - type: nauc_map_at_1_diff1 + value: 51.81510913391091 + - type: nauc_map_at_1_max + value: 31.45718877786518 + - type: nauc_map_at_1_std + value: 0.6652894870503548 + - type: nauc_map_at_20_diff1 + value: 44.362746077053586 + - type: nauc_map_at_20_max + value: 29.48842451907458 + - type: nauc_map_at_20_std + value: 2.1963216185045207 + - type: nauc_map_at_3_diff1 + value: 45.37627054912028 + - type: nauc_map_at_3_max + value: 28.883007725336995 + - type: nauc_map_at_3_std + value: -0.20173273127132102 + - type: nauc_map_at_5_diff1 + value: 44.268212156761955 + - type: nauc_map_at_5_max + value: 29.76691962328352 + - type: nauc_map_at_5_std + value: 1.0417698208020236 + - type: nauc_mrr_at_1000_diff1 + value: 44.21335029047463 + - type: nauc_mrr_at_1000_max + value: 30.963234867994643 + - type: nauc_mrr_at_1000_std + value: 3.208194178526169 + - type: nauc_mrr_at_100_diff1 + value: 44.18901172672336 + - type: nauc_mrr_at_100_max + value: 30.951281328812115 + - type: nauc_mrr_at_100_std + value: 3.21380311612713 + - type: nauc_mrr_at_10_diff1 + value: 44.07769644739813 + - type: nauc_mrr_at_10_max + value: 31.051393579292075 + - type: nauc_mrr_at_10_std + value: 3.0857305248011606 + - type: nauc_mrr_at_1_diff1 + value: 51.04332927712506 + - type: nauc_mrr_at_1_max + value: 33.10204278782818 + - type: nauc_mrr_at_1_std + value: 1.8848576137800481 + - type: nauc_mrr_at_20_diff1 + value: 44.09695747703678 + - type: nauc_mrr_at_20_max + value: 30.75215648042474 + - type: nauc_mrr_at_20_std + value: 3.0967110314793427 + - type: nauc_mrr_at_3_diff1 + value: 45.02671466831633 + - type: nauc_mrr_at_3_max + value: 31.14089473037222 + - type: nauc_mrr_at_3_std + value: 1.5549861369312517 + - type: nauc_mrr_at_5_diff1 + value: 44.01200575124768 + - type: nauc_mrr_at_5_max + value: 31.35251508067886 + - type: nauc_mrr_at_5_std + value: 2.436193105050964 + - type: nauc_ndcg_at_1000_diff1 + value: 42.98169967342506 + - type: nauc_ndcg_at_1000_max + value: 30.366039007252017 + - type: nauc_ndcg_at_1000_std + value: 5.049053046068428 + - type: nauc_ndcg_at_100_diff1 + value: 42.67834234453907 + - type: nauc_ndcg_at_100_max + value: 30.12269873054357 + - type: nauc_ndcg_at_100_std + value: 5.4827435194474115 + - type: nauc_ndcg_at_10_diff1 + value: 42.0598509029212 + - type: nauc_ndcg_at_10_max + value: 29.364390048123955 + - type: nauc_ndcg_at_10_std + value: 3.7649327138161643 + - type: nauc_ndcg_at_1_diff1 + value: 51.04332927712506 + - type: nauc_ndcg_at_1_max + value: 33.10204278782818 + - type: nauc_ndcg_at_1_std + value: 1.8848576137800481 + - type: nauc_ndcg_at_20_diff1 + value: 41.94853060381429 + - type: nauc_ndcg_at_20_max + value: 28.295786551394077 + - type: nauc_ndcg_at_20_std + value: 4.2132236763558994 + - type: nauc_ndcg_at_3_diff1 + value: 43.62007292783384 + - type: nauc_ndcg_at_3_max + value: 29.073529492941084 + - type: nauc_ndcg_at_3_std + value: -0.09275518285596841 + - type: nauc_ndcg_at_5_diff1 + value: 41.813267178694126 + - type: nauc_ndcg_at_5_max + value: 29.78237534055519 + - type: nauc_ndcg_at_5_std + value: 1.8127522501114646 + - type: nauc_precision_at_1000_diff1 + value: -13.48381165184421 + - type: nauc_precision_at_1000_max + value: -3.339018011199423 + - type: nauc_precision_at_1000_std + value: 7.608367277383381 + - type: nauc_precision_at_100_diff1 + value: 10.250769070048662 + - type: nauc_precision_at_100_max + value: 21.317873789567077 + - type: nauc_precision_at_100_std + value: 21.0333414305805 + - type: nauc_precision_at_10_diff1 + value: 27.757698966402184 + - type: nauc_precision_at_10_max + value: 29.44735713440695 + - type: nauc_precision_at_10_std + value: 13.064576002206083 + - type: nauc_precision_at_1_diff1 + value: 51.04332927712506 + - type: nauc_precision_at_1_max + value: 33.10204278782818 + - type: nauc_precision_at_1_std + value: 1.8848576137800481 + - type: nauc_precision_at_20_diff1 + value: 22.824655125705778 + - type: nauc_precision_at_20_max + value: 23.494370382164785 + - type: nauc_precision_at_20_std + value: 14.722644107210053 + - type: nauc_precision_at_3_diff1 + value: 36.090103514422 + - type: nauc_precision_at_3_max + value: 28.386051224276194 + - type: nauc_precision_at_3_std + value: 0.9968216658444929 + - type: nauc_precision_at_5_diff1 + value: 29.78267093767586 + - type: nauc_precision_at_5_max + value: 31.48300370842171 + - type: nauc_precision_at_5_std + value: 6.006559602847512 + - type: nauc_recall_at_1000_diff1 + value: 35.73868662162835 + - type: nauc_recall_at_1000_max + value: 48.383070976640724 + - type: nauc_recall_at_1000_std + value: 41.59587880417064 + - type: nauc_recall_at_100_diff1 + value: 34.28974657678655 + - type: nauc_recall_at_100_max + value: 29.896017976958777 + - type: nauc_recall_at_100_std + value: 21.02501453586133 + - type: nauc_recall_at_10_diff1 + value: 33.66338338383692 + - type: nauc_recall_at_10_max + value: 25.142317725047853 + - type: nauc_recall_at_10_std + value: 8.023904208965744 + - type: nauc_recall_at_1_diff1 + value: 51.81510913391091 + - type: nauc_recall_at_1_max + value: 31.45718877786518 + - type: nauc_recall_at_1_std + value: 0.6652894870503548 + - type: nauc_recall_at_20_diff1 + value: 32.673969436865235 + - type: nauc_recall_at_20_max + value: 20.01704582766808 + - type: nauc_recall_at_20_std + value: 9.941483488497981 + - type: nauc_recall_at_3_diff1 + value: 38.11222290105628 + - type: nauc_recall_at_3_max + value: 25.339532983105826 + - type: nauc_recall_at_3_std + value: -2.324451066363608 + - type: nauc_recall_at_5_diff1 + value: 33.63430870640167 + - type: nauc_recall_at_5_max + value: 26.791116604248906 + - type: nauc_recall_at_5_std + value: 2.4824568030362713 + - type: ndcg_at_1 + value: 29.020000000000003 + - type: ndcg_at_10 + value: 40.929 + - type: ndcg_at_100 + value: 46.022 + - type: ndcg_at_1000 + value: 48.25 + - type: ndcg_at_20 + value: 42.947 + - type: ndcg_at_3 + value: 36.124 + - type: ndcg_at_5 + value: 38.322 + - type: precision_at_1 + value: 29.020000000000003 + - type: precision_at_10 + value: 6.34 + - type: precision_at_100 + value: 0.9560000000000001 + - type: precision_at_1000 + value: 0.126 + - type: precision_at_20 + value: 3.66 + - type: precision_at_3 + value: 15.527 + - type: precision_at_5 + value: 10.721 + - type: recall_at_1 + value: 26.3 + - type: recall_at_10 + value: 54.49 + - type: recall_at_100 + value: 77.761 + - type: recall_at_1000 + value: 93.786 + - type: recall_at_20 + value: 62.114999999999995 + - type: recall_at_3 + value: 41.277 + - type: recall_at_5 + value: 46.711999999999996 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ClimateFEVER (default) + revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380 + split: test + type: mteb/climate-fever + metrics: + - type: main_score + value: 42.047000000000004 + - type: map_at_1 + value: 18.269 + - type: map_at_10 + value: 31.691999999999997 + - type: map_at_100 + value: 33.841 + - type: map_at_1000 + value: 34.009 + - type: map_at_20 + value: 32.885999999999996 + - type: map_at_3 + value: 26.558999999999997 + - type: map_at_5 + value: 29.119 + - type: mrr_at_1 + value: 41.69381107491857 + - type: mrr_at_10 + value: 54.293185460937984 + - type: mrr_at_100 + value: 54.87161862744807 + - type: mrr_at_1000 + value: 54.88722882443645 + - type: mrr_at_20 + value: 54.685754380853844 + - type: mrr_at_3 + value: 51.172638436482146 + - type: mrr_at_5 + value: 53.12052117263853 + - type: nauc_map_at_1000_diff1 + value: 27.03807365621228 + - type: nauc_map_at_1000_max + value: 40.31079671403445 + - type: nauc_map_at_1000_std + value: 26.092423798773883 + - type: nauc_map_at_100_diff1 + value: 27.015360474734436 + - type: nauc_map_at_100_max + value: 40.28408194597505 + - type: nauc_map_at_100_std + value: 26.086029014261968 + - type: nauc_map_at_10_diff1 + value: 27.222731203652856 + - type: nauc_map_at_10_max + value: 40.01781109904128 + - type: nauc_map_at_10_std + value: 24.73681887890272 + - type: nauc_map_at_1_diff1 + value: 35.75107300356484 + - type: nauc_map_at_1_max + value: 40.2201153742901 + - type: nauc_map_at_1_std + value: 18.766249947929374 + - type: nauc_map_at_20_diff1 + value: 26.931804653893042 + - type: nauc_map_at_20_max + value: 40.21587995014608 + - type: nauc_map_at_20_std + value: 25.75452108695598 + - type: nauc_map_at_3_diff1 + value: 28.310387788680696 + - type: nauc_map_at_3_max + value: 39.285866053656385 + - type: nauc_map_at_3_std + value: 21.394962842915703 + - type: nauc_map_at_5_diff1 + value: 27.300839773785274 + - type: nauc_map_at_5_max + value: 39.3888708340898 + - type: nauc_map_at_5_std + value: 22.78299335246201 + - type: nauc_mrr_at_1000_diff1 + value: 26.569029993582287 + - type: nauc_mrr_at_1000_max + value: 38.05698386072128 + - type: nauc_mrr_at_1000_std + value: 27.12877875031529 + - type: nauc_mrr_at_100_diff1 + value: 26.56693868451222 + - type: nauc_mrr_at_100_max + value: 38.06321319344823 + - type: nauc_mrr_at_100_std + value: 27.14409997788537 + - type: nauc_mrr_at_10_diff1 + value: 26.52694223161396 + - type: nauc_mrr_at_10_max + value: 38.120563154705 + - type: nauc_mrr_at_10_std + value: 27.11337497751667 + - type: nauc_mrr_at_1_diff1 + value: 29.371725407886277 + - type: nauc_mrr_at_1_max + value: 35.7850341702808 + - type: nauc_mrr_at_1_std + value: 22.69810863765783 + - type: nauc_mrr_at_20_diff1 + value: 26.567897033309247 + - type: nauc_mrr_at_20_max + value: 38.17484491649562 + - type: nauc_mrr_at_20_std + value: 27.218678564296972 + - type: nauc_mrr_at_3_diff1 + value: 26.582727973322427 + - type: nauc_mrr_at_3_max + value: 37.8745721692282 + - type: nauc_mrr_at_3_std + value: 26.567749469034307 + - type: nauc_mrr_at_5_diff1 + value: 26.404958533442898 + - type: nauc_mrr_at_5_max + value: 37.86090955141593 + - type: nauc_mrr_at_5_std + value: 26.65816459603454 + - type: nauc_ndcg_at_1000_diff1 + value: 25.7228323702742 + - type: nauc_ndcg_at_1000_max + value: 41.024272689913296 + - type: nauc_ndcg_at_1000_std + value: 31.373617783353815 + - type: nauc_ndcg_at_100_diff1 + value: 25.467806967471812 + - type: nauc_ndcg_at_100_max + value: 40.68595692225817 + - type: nauc_ndcg_at_100_std + value: 31.327255356351774 + - type: nauc_ndcg_at_10_diff1 + value: 25.65771458118311 + - type: nauc_ndcg_at_10_max + value: 40.2959313004829 + - type: nauc_ndcg_at_10_std + value: 28.21103387387833 + - type: nauc_ndcg_at_1_diff1 + value: 29.371725407886277 + - type: nauc_ndcg_at_1_max + value: 35.7850341702808 + - type: nauc_ndcg_at_1_std + value: 22.69810863765783 + - type: nauc_ndcg_at_20_diff1 + value: 25.008107221444327 + - type: nauc_ndcg_at_20_max + value: 40.613619354979626 + - type: nauc_ndcg_at_20_std + value: 30.216191744111416 + - type: nauc_ndcg_at_3_diff1 + value: 25.85227194113396 + - type: nauc_ndcg_at_3_max + value: 38.32492256264965 + - type: nauc_ndcg_at_3_std + value: 23.735358525961033 + - type: nauc_ndcg_at_5_diff1 + value: 25.747409532466243 + - type: nauc_ndcg_at_5_max + value: 39.4993084566524 + - type: nauc_ndcg_at_5_std + value: 25.19771375383721 + - type: nauc_precision_at_1000_diff1 + value: -8.149028290279253 + - type: nauc_precision_at_1000_max + value: -3.196086649201077 + - type: nauc_precision_at_1000_std + value: 13.643701012139948 + - type: nauc_precision_at_100_diff1 + value: -1.892485292157653 + - type: nauc_precision_at_100_max + value: 7.7434454354621245 + - type: nauc_precision_at_100_std + value: 22.988854451791806 + - type: nauc_precision_at_10_diff1 + value: 6.150550804828545 + - type: nauc_precision_at_10_max + value: 22.501131175285906 + - type: nauc_precision_at_10_std + value: 27.39677272392596 + - type: nauc_precision_at_1_diff1 + value: 29.371725407886277 + - type: nauc_precision_at_1_max + value: 35.7850341702808 + - type: nauc_precision_at_1_std + value: 22.69810863765783 + - type: nauc_precision_at_20_diff1 + value: 2.283445965946842 + - type: nauc_precision_at_20_max + value: 18.59466543059599 + - type: nauc_precision_at_20_std + value: 29.0738299597803 + - type: nauc_precision_at_3_diff1 + value: 12.963867454979258 + - type: nauc_precision_at_3_max + value: 30.449562657056333 + - type: nauc_precision_at_3_std + value: 25.581976194336352 + - type: nauc_precision_at_5_diff1 + value: 8.512947940252289 + - type: nauc_precision_at_5_max + value: 26.12425424420038 + - type: nauc_precision_at_5_std + value: 24.877415885322808 + - type: nauc_recall_at_1000_diff1 + value: 17.151717317242028 + - type: nauc_recall_at_1000_max + value: 40.67913325938115 + - type: nauc_recall_at_1000_std + value: 49.54837910314142 + - type: nauc_recall_at_100_diff1 + value: 16.83432440063162 + - type: nauc_recall_at_100_max + value: 34.46952489534257 + - type: nauc_recall_at_100_std + value: 38.26853671426454 + - type: nauc_recall_at_10_diff1 + value: 19.50239551179883 + - type: nauc_recall_at_10_max + value: 35.74261290262663 + - type: nauc_recall_at_10_std + value: 28.630457514118163 + - type: nauc_recall_at_1_diff1 + value: 35.75107300356484 + - type: nauc_recall_at_1_max + value: 40.2201153742901 + - type: nauc_recall_at_1_std + value: 18.766249947929374 + - type: nauc_recall_at_20_diff1 + value: 16.723000685755707 + - type: nauc_recall_at_20_max + value: 35.272383093342405 + - type: nauc_recall_at_20_std + value: 32.934757635631335 + - type: nauc_recall_at_3_diff1 + value: 24.024160029526794 + - type: nauc_recall_at_3_max + value: 38.07599046764463 + - type: nauc_recall_at_3_std + value: 22.648443171847685 + - type: nauc_recall_at_5_diff1 + value: 21.588763686113676 + - type: nauc_recall_at_5_max + value: 37.16237158404055 + - type: nauc_recall_at_5_std + value: 24.45061830715902 + - type: ndcg_at_1 + value: 41.693999999999996 + - type: ndcg_at_10 + value: 42.047000000000004 + - type: ndcg_at_100 + value: 49.309 + - type: ndcg_at_1000 + value: 51.861999999999995 + - type: ndcg_at_20 + value: 44.982 + - type: ndcg_at_3 + value: 35.510000000000005 + - type: ndcg_at_5 + value: 37.529 + - type: precision_at_1 + value: 41.693999999999996 + - type: precision_at_10 + value: 13.114 + - type: precision_at_100 + value: 2.1069999999999998 + - type: precision_at_1000 + value: 0.259 + - type: precision_at_20 + value: 7.824000000000001 + - type: precision_at_3 + value: 26.796999999999997 + - type: precision_at_5 + value: 20.169 + - type: recall_at_1 + value: 18.269 + - type: recall_at_10 + value: 48.44 + - type: recall_at_100 + value: 72.909 + - type: recall_at_1000 + value: 86.79400000000001 + - type: recall_at_20 + value: 56.714 + - type: recall_at_3 + value: 31.85 + - type: recall_at_5 + value: 38.488 + task: + type: Retrieval + - dataset: + config: default + name: MTEB DBPedia (default) + revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659 + split: test + type: mteb/dbpedia + metrics: + - type: main_score + value: 51.202000000000005 + - type: map_at_1 + value: 10.789 + - type: map_at_10 + value: 24.804000000000002 + - type: map_at_100 + value: 35.908 + - type: map_at_1000 + value: 37.97 + - type: map_at_20 + value: 29.236 + - type: map_at_3 + value: 17.055 + - type: map_at_5 + value: 20.342 + - type: mrr_at_1 + value: 78.0 + - type: mrr_at_10 + value: 83.82251984126987 + - type: mrr_at_100 + value: 84.00706659508124 + - type: mrr_at_1000 + value: 84.01172077534015 + - type: mrr_at_20 + value: 83.93946561479457 + - type: mrr_at_3 + value: 82.58333333333334 + - type: mrr_at_5 + value: 83.38333333333335 + - type: nauc_map_at_1000_diff1 + value: 21.975683577384412 + - type: nauc_map_at_1000_max + value: 33.104767603973286 + - type: nauc_map_at_1000_std + value: 19.507936661697688 + - type: nauc_map_at_100_diff1 + value: 23.19428288878281 + - type: nauc_map_at_100_max + value: 32.47043490749479 + - type: nauc_map_at_100_std + value: 16.611980248500473 + - type: nauc_map_at_10_diff1 + value: 23.314074413061576 + - type: nauc_map_at_10_max + value: 18.52506648908812 + - type: nauc_map_at_10_std + value: -9.718219448424597 + - type: nauc_map_at_1_diff1 + value: 27.402329635171146 + - type: nauc_map_at_1_max + value: 5.898746976402726 + - type: nauc_map_at_1_std + value: -26.703327281110212 + - type: nauc_map_at_20_diff1 + value: 23.613670514044472 + - type: nauc_map_at_20_max + value: 25.008187375084763 + - type: nauc_map_at_20_std + value: -0.05206367166066498 + - type: nauc_map_at_3_diff1 + value: 25.673374223753598 + - type: nauc_map_at_3_max + value: 12.527419567406877 + - type: nauc_map_at_3_std + value: -20.06963757181341 + - type: nauc_map_at_5_diff1 + value: 24.74002578400672 + - type: nauc_map_at_5_max + value: 14.23437788867648 + - type: nauc_map_at_5_std + value: -16.317803876665256 + - type: nauc_mrr_at_1000_diff1 + value: 53.26868100398232 + - type: nauc_mrr_at_1000_max + value: 67.65740877772801 + - type: nauc_mrr_at_1000_std + value: 39.43464159369656 + - type: nauc_mrr_at_100_diff1 + value: 53.25615896192901 + - type: nauc_mrr_at_100_max + value: 67.64777514366169 + - type: nauc_mrr_at_100_std + value: 39.410662043086006 + - type: nauc_mrr_at_10_diff1 + value: 52.94295111677663 + - type: nauc_mrr_at_10_max + value: 67.5393005296393 + - type: nauc_mrr_at_10_std + value: 39.31715440936177 + - type: nauc_mrr_at_1_diff1 + value: 57.148073445541826 + - type: nauc_mrr_at_1_max + value: 65.78742986970832 + - type: nauc_mrr_at_1_std + value: 34.198659989799246 + - type: nauc_mrr_at_20_diff1 + value: 53.223501273361265 + - type: nauc_mrr_at_20_max + value: 67.59762197314753 + - type: nauc_mrr_at_20_std + value: 39.359614957729356 + - type: nauc_mrr_at_3_diff1 + value: 53.619283112717184 + - type: nauc_mrr_at_3_max + value: 68.72067268448458 + - type: nauc_mrr_at_3_std + value: 40.53052904925793 + - type: nauc_mrr_at_5_diff1 + value: 52.86133436375577 + - type: nauc_mrr_at_5_max + value: 67.94415973414303 + - type: nauc_mrr_at_5_std + value: 40.09087346298919 + - type: nauc_ndcg_at_1000_diff1 + value: 31.008961737330505 + - type: nauc_ndcg_at_1000_max + value: 49.39127418414386 + - type: nauc_ndcg_at_1000_std + value: 37.509639671229806 + - type: nauc_ndcg_at_100_diff1 + value: 32.50484024525448 + - type: nauc_ndcg_at_100_max + value: 46.300662423725605 + - type: nauc_ndcg_at_100_std + value: 28.488771981297162 + - type: nauc_ndcg_at_10_diff1 + value: 27.911614286994414 + - type: nauc_ndcg_at_10_max + value: 44.70909339082426 + - type: nauc_ndcg_at_10_std + value: 25.644980583529154 + - type: nauc_ndcg_at_1_diff1 + value: 51.27342509891256 + - type: nauc_ndcg_at_1_max + value: 54.75803307782269 + - type: nauc_ndcg_at_1_std + value: 27.4853058050954 + - type: nauc_ndcg_at_20_diff1 + value: 30.29885920192407 + - type: nauc_ndcg_at_20_max + value: 43.45207612356715 + - type: nauc_ndcg_at_20_std + value: 21.59751863312113 + - type: nauc_ndcg_at_3_diff1 + value: 31.251071625533843 + - type: nauc_ndcg_at_3_max + value: 48.45180697571009 + - type: nauc_ndcg_at_3_std + value: 32.70662167853583 + - type: nauc_ndcg_at_5_diff1 + value: 26.175090671223877 + - type: nauc_ndcg_at_5_max + value: 45.2723355712432 + - type: nauc_ndcg_at_5_std + value: 31.461916393793 + - type: nauc_precision_at_1000_diff1 + value: -23.926082132378777 + - type: nauc_precision_at_1000_max + value: -9.350346667573811 + - type: nauc_precision_at_1000_std + value: 11.578726421051043 + - type: nauc_precision_at_100_diff1 + value: -7.468660956171794 + - type: nauc_precision_at_100_max + value: 19.470414434634723 + - type: nauc_precision_at_100_std + value: 43.86244545951367 + - type: nauc_precision_at_10_diff1 + value: -2.090265656696684 + - type: nauc_precision_at_10_max + value: 30.778228684745386 + - type: nauc_precision_at_10_std + value: 44.882546930240984 + - type: nauc_precision_at_1_diff1 + value: 57.148073445541826 + - type: nauc_precision_at_1_max + value: 65.78742986970832 + - type: nauc_precision_at_1_std + value: 34.198659989799246 + - type: nauc_precision_at_20_diff1 + value: -3.075798118380347 + - type: nauc_precision_at_20_max + value: 29.52951501638172 + - type: nauc_precision_at_20_std + value: 47.266521222769676 + - type: nauc_precision_at_3_diff1 + value: 11.892419680356198 + - type: nauc_precision_at_3_max + value: 43.146413741651415 + - type: nauc_precision_at_3_std + value: 45.2312022756118 + - type: nauc_precision_at_5_diff1 + value: 0.5765950918056327 + - type: nauc_precision_at_5_max + value: 34.22132902314228 + - type: nauc_precision_at_5_std + value: 44.78272426908718 + - type: nauc_recall_at_1000_diff1 + value: 24.99872069707702 + - type: nauc_recall_at_1000_max + value: 42.17319464089324 + - type: nauc_recall_at_1000_std + value: 47.42376725148043 + - type: nauc_recall_at_100_diff1 + value: 24.62929408109356 + - type: nauc_recall_at_100_max + value: 32.373805304406844 + - type: nauc_recall_at_100_std + value: 21.48682342485071 + - type: nauc_recall_at_10_diff1 + value: 20.62337020665992 + - type: nauc_recall_at_10_max + value: 14.125308316827395 + - type: nauc_recall_at_10_std + value: -13.565096294162865 + - type: nauc_recall_at_1_diff1 + value: 27.402329635171146 + - type: nauc_recall_at_1_max + value: 5.898746976402726 + - type: nauc_recall_at_1_std + value: -26.703327281110212 + - type: nauc_recall_at_20_diff1 + value: 22.169766882731277 + - type: nauc_recall_at_20_max + value: 20.588762488556828 + - type: nauc_recall_at_20_std + value: -4.530608772737279 + - type: nauc_recall_at_3_diff1 + value: 22.48622374174161 + - type: nauc_recall_at_3_max + value: 10.470407080375304 + - type: nauc_recall_at_3_std + value: -20.777479868757286 + - type: nauc_recall_at_5_diff1 + value: 21.28438252298866 + - type: nauc_recall_at_5_max + value: 10.424120660451583 + - type: nauc_recall_at_5_std + value: -17.912853638432384 + - type: ndcg_at_1 + value: 65.375 + - type: ndcg_at_10 + value: 51.202000000000005 + - type: ndcg_at_100 + value: 56.12200000000001 + - type: ndcg_at_1000 + value: 63.306 + - type: ndcg_at_20 + value: 50.442 + - type: ndcg_at_3 + value: 56.437000000000005 + - type: ndcg_at_5 + value: 53.861000000000004 + - type: precision_at_1 + value: 78.0 + - type: precision_at_10 + value: 41.075 + - type: precision_at_100 + value: 13.032 + - type: precision_at_1000 + value: 2.516 + - type: precision_at_20 + value: 31.4 + - type: precision_at_3 + value: 59.833000000000006 + - type: precision_at_5 + value: 51.9 + - type: recall_at_1 + value: 10.789 + - type: recall_at_10 + value: 30.059 + - type: recall_at_100 + value: 61.817 + - type: recall_at_1000 + value: 84.672 + - type: recall_at_20 + value: 39.135 + - type: recall_at_3 + value: 18.017 + - type: recall_at_5 + value: 22.492 + task: + type: Retrieval + - dataset: + config: default + name: MTEB EmotionClassification (default) + revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 + split: test + type: mteb/emotion + metrics: + - type: accuracy + value: 57.715 + - type: f1 + value: 51.85468544437296 + - type: f1_weighted + value: 58.73946069844862 + - type: main_score + value: 57.715 + task: + type: Classification + - dataset: + config: default + name: MTEB FEVER (default) + revision: bea83ef9e8fb933d90a2f1d5515737465d613e12 + split: test + type: mteb/fever + metrics: + - type: main_score + value: 92.438 + - type: map_at_1 + value: 82.678 + - type: map_at_10 + value: 89.90899999999999 + - type: map_at_100 + value: 90.09899999999999 + - type: map_at_1000 + value: 90.11 + - type: map_at_20 + value: 90.026 + - type: map_at_3 + value: 89.034 + - type: map_at_5 + value: 89.619 + - type: mrr_at_1 + value: 89.04890489048904 + - type: mrr_at_10 + value: 93.53417484605598 + - type: mrr_at_100 + value: 93.56969053658798 + - type: mrr_at_1000 + value: 93.56979354808294 + - type: mrr_at_20 + value: 93.56100677804474 + - type: mrr_at_3 + value: 93.25682568256818 + - type: mrr_at_5 + value: 93.46909690969086 + - type: nauc_map_at_1000_diff1 + value: 50.19087206783256 + - type: nauc_map_at_1000_max + value: 26.223996443425424 + - type: nauc_map_at_1000_std + value: -8.531486546405336 + - type: nauc_map_at_100_diff1 + value: 50.12601833237827 + - type: nauc_map_at_100_max + value: 26.205753684531942 + - type: nauc_map_at_100_std + value: -8.502300882475792 + - type: nauc_map_at_10_diff1 + value: 49.48962186883297 + - type: nauc_map_at_10_max + value: 25.849578028607546 + - type: nauc_map_at_10_std + value: -8.58622126027856 + - type: nauc_map_at_1_diff1 + value: 56.88016472114475 + - type: nauc_map_at_1_max + value: 24.671479435457048 + - type: nauc_map_at_1_std + value: -11.980878470985619 + - type: nauc_map_at_20_diff1 + value: 49.813384246326905 + - type: nauc_map_at_20_max + value: 25.96508257517373 + - type: nauc_map_at_20_std + value: -8.568670117647939 + - type: nauc_map_at_3_diff1 + value: 49.087764097890165 + - type: nauc_map_at_3_max + value: 25.65938258554376 + - type: nauc_map_at_3_std + value: -8.859093431924775 + - type: nauc_map_at_5_diff1 + value: 49.08166208415013 + - type: nauc_map_at_5_max + value: 25.696246071825684 + - type: nauc_map_at_5_std + value: -8.431713254517472 + - type: nauc_mrr_at_1000_diff1 + value: 73.35484368612293 + - type: nauc_mrr_at_1000_max + value: 35.657386688053336 + - type: nauc_mrr_at_1000_std + value: -18.09172713569766 + - type: nauc_mrr_at_100_diff1 + value: 73.35508125874483 + - type: nauc_mrr_at_100_max + value: 35.65842743437027 + - type: nauc_mrr_at_100_std + value: -18.08981699366641 + - type: nauc_mrr_at_10_diff1 + value: 73.29004337552368 + - type: nauc_mrr_at_10_max + value: 35.882001444609216 + - type: nauc_mrr_at_10_std + value: -18.05339396879553 + - type: nauc_mrr_at_1_diff1 + value: 74.48742882702338 + - type: nauc_mrr_at_1_max + value: 31.49138530538466 + - type: nauc_mrr_at_1_std + value: -19.510294856397955 + - type: nauc_mrr_at_20_diff1 + value: 73.3388656330962 + - type: nauc_mrr_at_20_max + value: 35.706948273788505 + - type: nauc_mrr_at_20_std + value: -18.140154123750992 + - type: nauc_mrr_at_3_diff1 + value: 73.22698350499 + - type: nauc_mrr_at_3_max + value: 36.4855373316516 + - type: nauc_mrr_at_3_std + value: -17.719256990311198 + - type: nauc_mrr_at_5_diff1 + value: 73.24460108538948 + - type: nauc_mrr_at_5_max + value: 36.322370705490634 + - type: nauc_mrr_at_5_std + value: -17.636279233457984 + - type: nauc_ndcg_at_1000_diff1 + value: 53.674109881592756 + - type: nauc_ndcg_at_1000_max + value: 28.767387846727487 + - type: nauc_ndcg_at_1000_std + value: -8.858681782014946 + - type: nauc_ndcg_at_100_diff1 + value: 52.33608078847966 + - type: nauc_ndcg_at_100_max + value: 28.511414384159877 + - type: nauc_ndcg_at_100_std + value: -8.085385430073922 + - type: nauc_ndcg_at_10_diff1 + value: 49.712295545440774 + - type: nauc_ndcg_at_10_max + value: 27.5674225152019 + - type: nauc_ndcg_at_10_std + value: -8.244677630275376 + - type: nauc_ndcg_at_1_diff1 + value: 74.48742882702338 + - type: nauc_ndcg_at_1_max + value: 31.49138530538466 + - type: nauc_ndcg_at_1_std + value: -19.510294856397955 + - type: nauc_ndcg_at_20_diff1 + value: 50.61628846813059 + - type: nauc_ndcg_at_20_max + value: 27.53989784238201 + - type: nauc_ndcg_at_20_std + value: -8.373695482986479 + - type: nauc_ndcg_at_3_diff1 + value: 51.295860863016884 + - type: nauc_ndcg_at_3_max + value: 28.99776689198307 + - type: nauc_ndcg_at_3_std + value: -8.878181909861983 + - type: nauc_ndcg_at_5_diff1 + value: 49.619081645734504 + - type: nauc_ndcg_at_5_max + value: 28.11109235395876 + - type: nauc_ndcg_at_5_std + value: -7.722157727171728 + - type: nauc_precision_at_1000_diff1 + value: -9.298540465937485 + - type: nauc_precision_at_1000_max + value: -1.3157308795912563 + - type: nauc_precision_at_1000_std + value: 1.897355386264135 + - type: nauc_precision_at_100_diff1 + value: -12.246672190804334 + - type: nauc_precision_at_100_max + value: -0.9687067276412682 + - type: nauc_precision_at_100_std + value: 4.56074518564851 + - type: nauc_precision_at_10_diff1 + value: -15.533411370200923 + - type: nauc_precision_at_10_max + value: -2.191843047666222 + - type: nauc_precision_at_10_std + value: 3.6723841478730748 + - type: nauc_precision_at_1_diff1 + value: 74.48742882702338 + - type: nauc_precision_at_1_max + value: 31.49138530538466 + - type: nauc_precision_at_1_std + value: -19.510294856397955 + - type: nauc_precision_at_20_diff1 + value: -15.290364061922347 + - type: nauc_precision_at_20_max + value: -2.921722171191804 + - type: nauc_precision_at_20_std + value: 4.08482465973661 + - type: nauc_precision_at_3_diff1 + value: -8.208906597107383 + - type: nauc_precision_at_3_max + value: 2.9796478961627284 + - type: nauc_precision_at_3_std + value: 0.34366033602604895 + - type: nauc_precision_at_5_diff1 + value: -14.42241522747573 + - type: nauc_precision_at_5_max + value: -0.5633890785935999 + - type: nauc_precision_at_5_std + value: 3.7064496791809836 + - type: nauc_recall_at_1000_diff1 + value: -0.5673198466803553 + - type: nauc_recall_at_1000_max + value: 21.92110385096128 + - type: nauc_recall_at_1000_std + value: 54.421987386115475 + - type: nauc_recall_at_100_diff1 + value: -0.6512704079314391 + - type: nauc_recall_at_100_max + value: 22.38252665262688 + - type: nauc_recall_at_100_std + value: 36.50750378730013 + - type: nauc_recall_at_10_diff1 + value: 11.308848658347774 + - type: nauc_recall_at_10_max + value: 21.077700181656738 + - type: nauc_recall_at_10_std + value: 8.321338697504787 + - type: nauc_recall_at_1_diff1 + value: 56.88016472114475 + - type: nauc_recall_at_1_max + value: 24.671479435457048 + - type: nauc_recall_at_1_std + value: -11.980878470985619 + - type: nauc_recall_at_20_diff1 + value: 5.8415071379210755 + - type: nauc_recall_at_20_max + value: 16.97886837481554 + - type: nauc_recall_at_20_std + value: 12.529145693495494 + - type: nauc_recall_at_3_diff1 + value: 27.396913234035086 + - type: nauc_recall_at_3_max + value: 24.897648442357994 + - type: nauc_recall_at_3_std + value: 0.8528297027573939 + - type: nauc_recall_at_5_diff1 + value: 18.295838017557397 + - type: nauc_recall_at_5_max + value: 24.077879268127823 + - type: nauc_recall_at_5_std + value: 7.099403908855888 + - type: ndcg_at_1 + value: 89.049 + - type: ndcg_at_10 + value: 92.438 + - type: ndcg_at_100 + value: 93.016 + - type: ndcg_at_1000 + value: 93.17699999999999 + - type: ndcg_at_20 + value: 92.713 + - type: ndcg_at_3 + value: 91.40599999999999 + - type: ndcg_at_5 + value: 92.026 + - type: precision_at_1 + value: 89.049 + - type: precision_at_10 + value: 10.917 + - type: precision_at_100 + value: 1.146 + - type: precision_at_1000 + value: 0.117 + - type: precision_at_20 + value: 5.56 + - type: precision_at_3 + value: 34.598 + - type: precision_at_5 + value: 21.323 + - type: recall_at_1 + value: 82.678 + - type: recall_at_10 + value: 96.465 + - type: recall_at_100 + value: 98.571 + - type: recall_at_1000 + value: 99.496 + - type: recall_at_20 + value: 97.342 + - type: recall_at_3 + value: 93.696 + - type: recall_at_5 + value: 95.324 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FiQA2018 (default) + revision: 27a168819829fe9bcd655c2df245fb19452e8e06 + split: test + type: mteb/fiqa + metrics: + - type: main_score + value: 58.781000000000006 + - type: map_at_1 + value: 31.107000000000003 + - type: map_at_10 + value: 50.955 + - type: map_at_100 + value: 53.177 + - type: map_at_1000 + value: 53.291 + - type: map_at_20 + value: 52.271 + - type: map_at_3 + value: 44.762 + - type: map_at_5 + value: 48.379 + - type: mrr_at_1 + value: 58.48765432098766 + - type: mrr_at_10 + value: 66.11429551244368 + - type: mrr_at_100 + value: 66.68929754431386 + - type: mrr_at_1000 + value: 66.71304995006113 + - type: mrr_at_20 + value: 66.46821550477237 + - type: mrr_at_3 + value: 64.22325102880657 + - type: mrr_at_5 + value: 65.41152263374484 + - type: nauc_map_at_1000_diff1 + value: 45.490146083064445 + - type: nauc_map_at_1000_max + value: 33.573139354617126 + - type: nauc_map_at_1000_std + value: -14.07140489937541 + - type: nauc_map_at_100_diff1 + value: 45.48828357408913 + - type: nauc_map_at_100_max + value: 33.51907944260763 + - type: nauc_map_at_100_std + value: -14.059609883903152 + - type: nauc_map_at_10_diff1 + value: 45.70748844526757 + - type: nauc_map_at_10_max + value: 31.667587503334587 + - type: nauc_map_at_10_std + value: -15.076948336390855 + - type: nauc_map_at_1_diff1 + value: 51.42775649850064 + - type: nauc_map_at_1_max + value: 16.56862308325116 + - type: nauc_map_at_1_std + value: -14.684731980257675 + - type: nauc_map_at_20_diff1 + value: 45.754998522906284 + - type: nauc_map_at_20_max + value: 33.03759060247343 + - type: nauc_map_at_20_std + value: -14.750787459968736 + - type: nauc_map_at_3_diff1 + value: 46.45241223088609 + - type: nauc_map_at_3_max + value: 26.607789112226467 + - type: nauc_map_at_3_std + value: -14.997049792585598 + - type: nauc_map_at_5_diff1 + value: 45.87702900983919 + - type: nauc_map_at_5_max + value: 30.076255479914348 + - type: nauc_map_at_5_std + value: -15.062787509367553 + - type: nauc_mrr_at_1000_diff1 + value: 55.64889336097758 + - type: nauc_mrr_at_1000_max + value: 48.57022261913911 + - type: nauc_mrr_at_1000_std + value: -12.428435474800143 + - type: nauc_mrr_at_100_diff1 + value: 55.62957328562593 + - type: nauc_mrr_at_100_max + value: 48.56575267775789 + - type: nauc_mrr_at_100_std + value: -12.415226616847987 + - type: nauc_mrr_at_10_diff1 + value: 55.5931002027865 + - type: nauc_mrr_at_10_max + value: 48.428200063552374 + - type: nauc_mrr_at_10_std + value: -12.590361961152267 + - type: nauc_mrr_at_1_diff1 + value: 59.470635489729105 + - type: nauc_mrr_at_1_max + value: 49.66866699872627 + - type: nauc_mrr_at_1_std + value: -13.590112604913607 + - type: nauc_mrr_at_20_diff1 + value: 55.60145155716686 + - type: nauc_mrr_at_20_max + value: 48.58677663675733 + - type: nauc_mrr_at_20_std + value: -12.454093344399036 + - type: nauc_mrr_at_3_diff1 + value: 55.76657118158415 + - type: nauc_mrr_at_3_max + value: 48.88547787372198 + - type: nauc_mrr_at_3_std + value: -13.299744066289124 + - type: nauc_mrr_at_5_diff1 + value: 55.55217612731964 + - type: nauc_mrr_at_5_max + value: 48.56957852769844 + - type: nauc_mrr_at_5_std + value: -12.876904435466624 + - type: nauc_ndcg_at_1000_diff1 + value: 47.2645656074121 + - type: nauc_ndcg_at_1000_max + value: 39.95808937564202 + - type: nauc_ndcg_at_1000_std + value: -11.366829207572232 + - type: nauc_ndcg_at_100_diff1 + value: 46.89043419464991 + - type: nauc_ndcg_at_100_max + value: 39.00034359981605 + - type: nauc_ndcg_at_100_std + value: -10.697277437129921 + - type: nauc_ndcg_at_10_diff1 + value: 47.07625032910763 + - type: nauc_ndcg_at_10_max + value: 35.51275239983428 + - type: nauc_ndcg_at_10_std + value: -13.965305287946128 + - type: nauc_ndcg_at_1_diff1 + value: 59.470635489729105 + - type: nauc_ndcg_at_1_max + value: 49.66866699872627 + - type: nauc_ndcg_at_1_std + value: -13.590112604913607 + - type: nauc_ndcg_at_20_diff1 + value: 47.44262917418296 + - type: nauc_ndcg_at_20_max + value: 37.6804112715633 + - type: nauc_ndcg_at_20_std + value: -13.174880813005297 + - type: nauc_ndcg_at_3_diff1 + value: 44.56982475937759 + - type: nauc_ndcg_at_3_max + value: 37.96424549723314 + - type: nauc_ndcg_at_3_std + value: -13.657607148249964 + - type: nauc_ndcg_at_5_diff1 + value: 45.427291740214024 + - type: nauc_ndcg_at_5_max + value: 35.42232275517991 + - type: nauc_ndcg_at_5_std + value: -14.510048307634808 + - type: nauc_precision_at_1000_diff1 + value: -16.58479747595096 + - type: nauc_precision_at_1000_max + value: 27.22386867486023 + - type: nauc_precision_at_1000_std + value: 9.41210384044254 + - type: nauc_precision_at_100_diff1 + value: -11.640382840009572 + - type: nauc_precision_at_100_max + value: 30.20752947841474 + - type: nauc_precision_at_100_std + value: 10.72773947232612 + - type: nauc_precision_at_10_diff1 + value: 3.2540578244055594 + - type: nauc_precision_at_10_max + value: 35.80515547017638 + - type: nauc_precision_at_10_std + value: 0.299517152086918 + - type: nauc_precision_at_1_diff1 + value: 59.470635489729105 + - type: nauc_precision_at_1_max + value: 49.66866699872627 + - type: nauc_precision_at_1_std + value: -13.590112604913607 + - type: nauc_precision_at_20_diff1 + value: -1.8627219860435185 + - type: nauc_precision_at_20_max + value: 35.9181314633325 + - type: nauc_precision_at_20_std + value: 4.491869749000042 + - type: nauc_precision_at_3_diff1 + value: 17.94168903901189 + - type: nauc_precision_at_3_max + value: 41.67388438464254 + - type: nauc_precision_at_3_std + value: -5.38615084998387 + - type: nauc_precision_at_5_diff1 + value: 9.312012525324068 + - type: nauc_precision_at_5_max + value: 39.52463080415461 + - type: nauc_precision_at_5_std + value: -2.615286156278468 + - type: nauc_recall_at_1000_diff1 + value: 37.0960616996064 + - type: nauc_recall_at_1000_max + value: 46.91967503624078 + - type: nauc_recall_at_1000_std + value: 36.70723581015844 + - type: nauc_recall_at_100_diff1 + value: 32.54497560045993 + - type: nauc_recall_at_100_max + value: 26.846226776082734 + - type: nauc_recall_at_100_std + value: 9.257918182671672 + - type: nauc_recall_at_10_diff1 + value: 40.05619869408745 + - type: nauc_recall_at_10_max + value: 25.504319960057014 + - type: nauc_recall_at_10_std + value: -12.57012842016253 + - type: nauc_recall_at_1_diff1 + value: 51.42775649850064 + - type: nauc_recall_at_1_max + value: 16.56862308325116 + - type: nauc_recall_at_1_std + value: -14.684731980257675 + - type: nauc_recall_at_20_diff1 + value: 39.34128607816815 + - type: nauc_recall_at_20_max + value: 28.31147877410395 + - type: nauc_recall_at_20_std + value: -10.295180225906224 + - type: nauc_recall_at_3_diff1 + value: 41.31333745355922 + - type: nauc_recall_at_3_max + value: 22.642649370921276 + - type: nauc_recall_at_3_std + value: -14.44811859378254 + - type: nauc_recall_at_5_diff1 + value: 39.91795256714951 + - type: nauc_recall_at_5_max + value: 24.396817798634245 + - type: nauc_recall_at_5_std + value: -13.696077909471175 + - type: ndcg_at_1 + value: 58.48799999999999 + - type: ndcg_at_10 + value: 58.781000000000006 + - type: ndcg_at_100 + value: 65.212 + - type: ndcg_at_1000 + value: 66.85900000000001 + - type: ndcg_at_20 + value: 61.529999999999994 + - type: ndcg_at_3 + value: 54.864000000000004 + - type: ndcg_at_5 + value: 56.223 + - type: precision_at_1 + value: 58.48799999999999 + - type: precision_at_10 + value: 16.111 + - type: precision_at_100 + value: 2.298 + - type: precision_at_1000 + value: 0.257 + - type: precision_at_20 + value: 9.306000000000001 + - type: precision_at_3 + value: 36.317 + - type: precision_at_5 + value: 26.759 + - type: recall_at_1 + value: 31.107000000000003 + - type: recall_at_10 + value: 65.08500000000001 + - type: recall_at_100 + value: 87.91 + - type: recall_at_1000 + value: 97.817 + - type: recall_at_20 + value: 73.282 + - type: recall_at_3 + value: 49.317 + - type: recall_at_5 + value: 56.617 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HotpotQA (default) + revision: ab518f4d6fcca38d87c25209f94beba119d02014 + split: test + type: mteb/hotpotqa + metrics: + - type: main_score + value: 79.449 + - type: map_at_1 + value: 41.384 + - type: map_at_10 + value: 72.844 + - type: map_at_100 + value: 73.589 + - type: map_at_1000 + value: 73.624 + - type: map_at_20 + value: 73.317 + - type: map_at_3 + value: 69.427 + - type: map_at_5 + value: 71.68299999999999 + - type: mrr_at_1 + value: 82.76839972991222 + - type: mrr_at_10 + value: 87.92045807744634 + - type: mrr_at_100 + value: 88.05529589670978 + - type: mrr_at_1000 + value: 88.05935891074716 + - type: mrr_at_20 + value: 88.01402112970962 + - type: mrr_at_3 + value: 87.3216295295969 + - type: mrr_at_5 + value: 87.69705154175074 + - type: nauc_map_at_1000_diff1 + value: 20.396629535039704 + - type: nauc_map_at_1000_max + value: 39.10949908339265 + - type: nauc_map_at_1000_std + value: 10.224729673688502 + - type: nauc_map_at_100_diff1 + value: 20.381077063965574 + - type: nauc_map_at_100_max + value: 39.12262980169527 + - type: nauc_map_at_100_std + value: 10.256952440972226 + - type: nauc_map_at_10_diff1 + value: 20.227214487416916 + - type: nauc_map_at_10_max + value: 39.065878364926085 + - type: nauc_map_at_10_std + value: 9.830819360569484 + - type: nauc_map_at_1_diff1 + value: 60.61929089121275 + - type: nauc_map_at_1_max + value: 49.53547409224507 + - type: nauc_map_at_1_std + value: 0.2722096857291782 + - type: nauc_map_at_20_diff1 + value: 20.183915365850165 + - type: nauc_map_at_20_max + value: 39.06905710390586 + - type: nauc_map_at_20_std + value: 10.244769286257812 + - type: nauc_map_at_3_diff1 + value: 18.953350220177363 + - type: nauc_map_at_3_max + value: 36.89647666189664 + - type: nauc_map_at_3_std + value: 6.856939205711613 + - type: nauc_map_at_5_diff1 + value: 19.74313508534105 + - type: nauc_map_at_5_max + value: 38.42860611762909 + - type: nauc_map_at_5_std + value: 8.620757357067802 + - type: nauc_mrr_at_1000_diff1 + value: 60.544760748070416 + - type: nauc_mrr_at_1000_max + value: 53.536700750600176 + - type: nauc_mrr_at_1000_std + value: 4.356103341419562 + - type: nauc_mrr_at_100_diff1 + value: 60.543037850402825 + - type: nauc_mrr_at_100_max + value: 53.54473925679791 + - type: nauc_mrr_at_100_std + value: 4.3713759172294475 + - type: nauc_mrr_at_10_diff1 + value: 60.57585979923885 + - type: nauc_mrr_at_10_max + value: 53.65882404973961 + - type: nauc_mrr_at_10_std + value: 4.46866142907982 + - type: nauc_mrr_at_1_diff1 + value: 60.61929089121275 + - type: nauc_mrr_at_1_max + value: 49.53547409224507 + - type: nauc_mrr_at_1_std + value: 0.2722096857291782 + - type: nauc_mrr_at_20_diff1 + value: 60.541893232518674 + - type: nauc_mrr_at_20_max + value: 53.6135776399171 + - type: nauc_mrr_at_20_std + value: 4.443552945861195 + - type: nauc_mrr_at_3_diff1 + value: 60.46996364153697 + - type: nauc_mrr_at_3_max + value: 53.981024588336936 + - type: nauc_mrr_at_3_std + value: 4.300285863686253 + - type: nauc_mrr_at_5_diff1 + value: 60.562791070200426 + - type: nauc_mrr_at_5_max + value: 53.884058343579966 + - type: nauc_mrr_at_5_std + value: 4.35333313705802 + - type: nauc_ndcg_at_1000_diff1 + value: 26.909558826785485 + - type: nauc_ndcg_at_1000_max + value: 43.2090252545764 + - type: nauc_ndcg_at_1000_std + value: 13.24632397019833 + - type: nauc_ndcg_at_100_diff1 + value: 26.4096138903785 + - type: nauc_ndcg_at_100_max + value: 43.50667894420325 + - type: nauc_ndcg_at_100_std + value: 14.272929786830657 + - type: nauc_ndcg_at_10_diff1 + value: 25.261392560708607 + - type: nauc_ndcg_at_10_max + value: 43.02496845139645 + - type: nauc_ndcg_at_10_std + value: 12.753991213996402 + - type: nauc_ndcg_at_1_diff1 + value: 60.61929089121275 + - type: nauc_ndcg_at_1_max + value: 49.53547409224507 + - type: nauc_ndcg_at_1_std + value: 0.2722096857291782 + - type: nauc_ndcg_at_20_diff1 + value: 25.15730629354081 + - type: nauc_ndcg_at_20_max + value: 43.10358742768409 + - type: nauc_ndcg_at_20_std + value: 14.103247675055986 + - type: nauc_ndcg_at_3_diff1 + value: 23.492158440363873 + - type: nauc_ndcg_at_3_max + value: 39.880317429264736 + - type: nauc_ndcg_at_3_std + value: 7.852278799949863 + - type: nauc_ndcg_at_5_diff1 + value: 24.46471897598423 + - type: nauc_ndcg_at_5_max + value: 41.901821932685294 + - type: nauc_ndcg_at_5_std + value: 10.33482164145028 + - type: nauc_precision_at_1000_diff1 + value: 14.556112531859444 + - type: nauc_precision_at_1000_max + value: 54.51236512101235 + - type: nauc_precision_at_1000_std + value: 68.89420216988455 + - type: nauc_precision_at_100_diff1 + value: 14.116319404924122 + - type: nauc_precision_at_100_max + value: 50.42943334977378 + - type: nauc_precision_at_100_std + value: 49.80016017936658 + - type: nauc_precision_at_10_diff1 + value: 14.530495877243805 + - type: nauc_precision_at_10_max + value: 43.89651175033577 + - type: nauc_precision_at_10_std + value: 24.764789718434958 + - type: nauc_precision_at_1_diff1 + value: 60.61929089121275 + - type: nauc_precision_at_1_max + value: 49.53547409224507 + - type: nauc_precision_at_1_std + value: 0.2722096857291782 + - type: nauc_precision_at_20_diff1 + value: 11.499635650364958 + - type: nauc_precision_at_20_max + value: 44.499499741252265 + - type: nauc_precision_at_20_std + value: 33.743842605352725 + - type: nauc_precision_at_3_diff1 + value: 14.621019803797811 + - type: nauc_precision_at_3_max + value: 38.1391146398071 + - type: nauc_precision_at_3_std + value: 11.050680597126348 + - type: nauc_precision_at_5_diff1 + value: 14.878056511475538 + - type: nauc_precision_at_5_max + value: 41.52854585813069 + - type: nauc_precision_at_5_std + value: 16.596884488946877 + - type: nauc_recall_at_1000_diff1 + value: 14.556112531860405 + - type: nauc_recall_at_1000_max + value: 54.512365121012444 + - type: nauc_recall_at_1000_std + value: 68.89420216988472 + - type: nauc_recall_at_100_diff1 + value: 14.11631940492389 + - type: nauc_recall_at_100_max + value: 50.42943334977325 + - type: nauc_recall_at_100_std + value: 49.80016017936635 + - type: nauc_recall_at_10_diff1 + value: 14.530495877243975 + - type: nauc_recall_at_10_max + value: 43.89651175033581 + - type: nauc_recall_at_10_std + value: 24.764789718434855 + - type: nauc_recall_at_1_diff1 + value: 60.61929089121275 + - type: nauc_recall_at_1_max + value: 49.53547409224507 + - type: nauc_recall_at_1_std + value: 0.2722096857291782 + - type: nauc_recall_at_20_diff1 + value: 11.499635650364953 + - type: nauc_recall_at_20_max + value: 44.499499741252166 + - type: nauc_recall_at_20_std + value: 33.74384260535269 + - type: nauc_recall_at_3_diff1 + value: 14.621019803797758 + - type: nauc_recall_at_3_max + value: 38.139114639807104 + - type: nauc_recall_at_3_std + value: 11.050680597126208 + - type: nauc_recall_at_5_diff1 + value: 14.87805651147543 + - type: nauc_recall_at_5_max + value: 41.52854585813069 + - type: nauc_recall_at_5_std + value: 16.59688448894684 + - type: ndcg_at_1 + value: 82.768 + - type: ndcg_at_10 + value: 79.449 + - type: ndcg_at_100 + value: 81.878 + - type: ndcg_at_1000 + value: 82.526 + - type: ndcg_at_20 + value: 80.601 + - type: ndcg_at_3 + value: 74.899 + - type: ndcg_at_5 + value: 77.586 + - type: precision_at_1 + value: 82.768 + - type: precision_at_10 + value: 16.804 + - type: precision_at_100 + value: 1.8659999999999999 + - type: precision_at_1000 + value: 0.19499999999999998 + - type: precision_at_20 + value: 8.770999999999999 + - type: precision_at_3 + value: 49.381 + - type: precision_at_5 + value: 31.746000000000002 + - type: recall_at_1 + value: 41.384 + - type: recall_at_10 + value: 84.018 + - type: recall_at_100 + value: 93.30199999999999 + - type: recall_at_1000 + value: 97.529 + - type: recall_at_20 + value: 87.711 + - type: recall_at_3 + value: 74.072 + - type: recall_at_5 + value: 79.365 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ImdbClassification (default) + revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 + split: test + type: mteb/imdb + metrics: + - type: accuracy + value: 95.50119999999997 + - type: ap + value: 93.27855740989341 + - type: ap_weighted + value: 93.27855740989341 + - type: f1 + value: 95.49922732391366 + - type: f1_weighted + value: 95.49922732391366 + - type: main_score + value: 95.50119999999997 + task: + type: Classification + - dataset: + config: default + name: MTEB MSMARCO (default) + revision: c5a29a104738b98a9e76336939199e264163d4a0 + split: dev + type: mteb/msmarco + metrics: + - type: main_score + value: 44.181 + - type: map_at_1 + value: 24.3 + - type: map_at_10 + value: 37.064 + - type: map_at_100 + value: 38.217 + - type: map_at_1000 + value: 38.261 + - type: map_at_20 + value: 37.797 + - type: map_at_3 + value: 33.03 + - type: map_at_5 + value: 35.382000000000005 + - type: mrr_at_1 + value: 25.014326647564474 + - type: mrr_at_10 + value: 37.67002092145362 + - type: mrr_at_100 + value: 38.76618716955713 + - type: mrr_at_1000 + value: 38.803895343578624 + - type: mrr_at_20 + value: 38.372875531879025 + - type: mrr_at_3 + value: 33.74164278892073 + - type: mrr_at_5 + value: 36.04250238777461 + - type: nauc_map_at_1000_diff1 + value: 37.38914109165067 + - type: nauc_map_at_1000_max + value: 9.290439022090213 + - type: nauc_map_at_1000_std + value: -17.68507604596775 + - type: nauc_map_at_100_diff1 + value: 37.3858106261435 + - type: nauc_map_at_100_max + value: 9.292194370842791 + - type: nauc_map_at_100_std + value: -17.6461510679294 + - type: nauc_map_at_10_diff1 + value: 37.24355836056403 + - type: nauc_map_at_10_max + value: 9.19029394636661 + - type: nauc_map_at_10_std + value: -18.216369315567626 + - type: nauc_map_at_1_diff1 + value: 40.298938486026984 + - type: nauc_map_at_1_max + value: 8.149499405622326 + - type: nauc_map_at_1_std + value: -17.09168853307602 + - type: nauc_map_at_20_diff1 + value: 37.344123641575216 + - type: nauc_map_at_20_max + value: 9.24559383901809 + - type: nauc_map_at_20_std + value: -17.842740773642962 + - type: nauc_map_at_3_diff1 + value: 37.4023127968177 + - type: nauc_map_at_3_max + value: 8.930674077317596 + - type: nauc_map_at_3_std + value: -18.68520909934096 + - type: nauc_map_at_5_diff1 + value: 37.12600186091895 + - type: nauc_map_at_5_max + value: 9.173506919924861 + - type: nauc_map_at_5_std + value: -18.625677130615294 + - type: nauc_mrr_at_1000_diff1 + value: 37.34256456294692 + - type: nauc_mrr_at_1000_max + value: 9.276741130450404 + - type: nauc_mrr_at_1000_std + value: -17.41693013754444 + - type: nauc_mrr_at_100_diff1 + value: 37.33775949993714 + - type: nauc_mrr_at_100_max + value: 9.28051163202218 + - type: nauc_mrr_at_100_std + value: -17.381741706111445 + - type: nauc_mrr_at_10_diff1 + value: 37.21506505847139 + - type: nauc_mrr_at_10_max + value: 9.200324529184542 + - type: nauc_mrr_at_10_std + value: -17.904321523440817 + - type: nauc_mrr_at_1_diff1 + value: 40.314678345050915 + - type: nauc_mrr_at_1_max + value: 8.193685362111243 + - type: nauc_mrr_at_1_std + value: -17.096535887474175 + - type: nauc_mrr_at_20_diff1 + value: 37.293746882874004 + - type: nauc_mrr_at_20_max + value: 9.256273923676206 + - type: nauc_mrr_at_20_std + value: -17.528338232043577 + - type: nauc_mrr_at_3_diff1 + value: 37.254812254578376 + - type: nauc_mrr_at_3_max + value: 8.903676300128614 + - type: nauc_mrr_at_3_std + value: -18.49940979312031 + - type: nauc_mrr_at_5_diff1 + value: 37.08969825523026 + - type: nauc_mrr_at_5_max + value: 9.194982897416688 + - type: nauc_mrr_at_5_std + value: -18.291840579141315 + - type: nauc_ndcg_at_1000_diff1 + value: 36.930810397557096 + - type: nauc_ndcg_at_1000_max + value: 9.8356345032183 + - type: nauc_ndcg_at_1000_std + value: -16.308145152943887 + - type: nauc_ndcg_at_100_diff1 + value: 36.901149744427414 + - type: nauc_ndcg_at_100_max + value: 9.96065454342114 + - type: nauc_ndcg_at_100_std + value: -14.983815239399584 + - type: nauc_ndcg_at_10_diff1 + value: 36.441571794416724 + - type: nauc_ndcg_at_10_max + value: 9.57337658776914 + - type: nauc_ndcg_at_10_std + value: -17.88037638294921 + - type: nauc_ndcg_at_1_diff1 + value: 40.314678345050915 + - type: nauc_ndcg_at_1_max + value: 8.193685362111243 + - type: nauc_ndcg_at_1_std + value: -17.096535887474175 + - type: nauc_ndcg_at_20_diff1 + value: 36.775334219857484 + - type: nauc_ndcg_at_20_max + value: 9.789544462660507 + - type: nauc_ndcg_at_20_std + value: -16.465733594062474 + - type: nauc_ndcg_at_3_diff1 + value: 36.58838956901628 + - type: nauc_ndcg_at_3_max + value: 9.089768089567865 + - type: nauc_ndcg_at_3_std + value: -19.12823913473232 + - type: nauc_ndcg_at_5_diff1 + value: 36.147729725463364 + - type: nauc_ndcg_at_5_max + value: 9.53707003144017 + - type: nauc_ndcg_at_5_std + value: -18.91372487441106 + - type: nauc_precision_at_1000_diff1 + value: -6.013504255890098 + - type: nauc_precision_at_1000_max + value: 6.319348588937731 + - type: nauc_precision_at_1000_std + value: 6.360339202992953 + - type: nauc_precision_at_100_diff1 + value: 14.846649240680357 + - type: nauc_precision_at_100_max + value: 11.751644343520605 + - type: nauc_precision_at_100_std + value: 16.881205928162444 + - type: nauc_precision_at_10_diff1 + value: 30.328513184776966 + - type: nauc_precision_at_10_max + value: 9.988509735977631 + - type: nauc_precision_at_10_std + value: -15.609966599969837 + - type: nauc_precision_at_1_diff1 + value: 40.314678345050915 + - type: nauc_precision_at_1_max + value: 8.193685362111243 + - type: nauc_precision_at_1_std + value: -17.096535887474175 + - type: nauc_precision_at_20_diff1 + value: 28.248245250811543 + - type: nauc_precision_at_20_max + value: 10.953279209883918 + - type: nauc_precision_at_20_std + value: -7.365540710727016 + - type: nauc_precision_at_3_diff1 + value: 33.6150964111514 + - type: nauc_precision_at_3_max + value: 9.216455510763346 + - type: nauc_precision_at_3_std + value: -20.45932513010908 + - type: nauc_precision_at_5_diff1 + value: 31.518755311864705 + - type: nauc_precision_at_5_max + value: 10.019710006442747 + - type: nauc_precision_at_5_std + value: -19.740528698385468 + - type: nauc_recall_at_1000_diff1 + value: 12.207155589507542 + - type: nauc_recall_at_1000_max + value: 39.3447783153665 + - type: nauc_recall_at_1000_std + value: 74.60352827999826 + - type: nauc_recall_at_100_diff1 + value: 32.993666280768615 + - type: nauc_recall_at_100_max + value: 16.487188889720816 + - type: nauc_recall_at_100_std + value: 26.828206265371275 + - type: nauc_recall_at_10_diff1 + value: 33.65453771237772 + - type: nauc_recall_at_10_max + value: 10.71869814574723 + - type: nauc_recall_at_10_std + value: -16.27859785753318 + - type: nauc_recall_at_1_diff1 + value: 40.298938486026984 + - type: nauc_recall_at_1_max + value: 8.149499405622326 + - type: nauc_recall_at_1_std + value: -17.09168853307602 + - type: nauc_recall_at_20_diff1 + value: 34.60034971417269 + - type: nauc_recall_at_20_max + value: 12.076871992384788 + - type: nauc_recall_at_20_std + value: -8.224571589978806 + - type: nauc_recall_at_3_diff1 + value: 34.24661417034744 + - type: nauc_recall_at_3_max + value: 9.464103325281997 + - type: nauc_recall_at_3_std + value: -20.329748455626195 + - type: nauc_recall_at_5_diff1 + value: 33.042225241281585 + - type: nauc_recall_at_5_max + value: 10.486814885646142 + - type: nauc_recall_at_5_std + value: -19.7259662900716 + - type: ndcg_at_1 + value: 25.013999999999996 + - type: ndcg_at_10 + value: 44.181 + - type: ndcg_at_100 + value: 49.673 + - type: ndcg_at_1000 + value: 50.705999999999996 + - type: ndcg_at_20 + value: 46.798 + - type: ndcg_at_3 + value: 36.037 + - type: ndcg_at_5 + value: 40.214 + - type: precision_at_1 + value: 25.013999999999996 + - type: precision_at_10 + value: 6.9110000000000005 + - type: precision_at_100 + value: 0.9650000000000001 + - type: precision_at_1000 + value: 0.105 + - type: precision_at_20 + value: 4.004 + - type: precision_at_3 + value: 15.238999999999999 + - type: precision_at_5 + value: 11.264000000000001 + - type: recall_at_1 + value: 24.3 + - type: recall_at_10 + value: 66.06400000000001 + - type: recall_at_100 + value: 91.291 + - type: recall_at_1000 + value: 99.054 + - type: recall_at_20 + value: 76.25699999999999 + - type: recall_at_3 + value: 44.039 + - type: recall_at_5 + value: 54.053 + task: + type: Retrieval + - dataset: + config: default + name: MTEB MSMARCO (default) + revision: c5a29a104738b98a9e76336939199e264163d4a0 + split: test + type: mteb/msmarco + metrics: + - type: main_score + value: 74.044 + - type: map_at_1 + value: 2.307 + - type: map_at_10 + value: 16.837 + - type: map_at_100 + value: 45.499 + - type: map_at_1000 + value: 55.629 + - type: map_at_20 + value: 27.256999999999998 + - type: map_at_3 + value: 6.068 + - type: map_at_5 + value: 9.496 + - type: mrr_at_1 + value: 93.02325581395348 + - type: mrr_at_10 + value: 96.12403100775194 + - type: mrr_at_100 + value: 96.12403100775194 + - type: mrr_at_1000 + value: 96.12403100775194 + - type: mrr_at_20 + value: 96.12403100775194 + - type: mrr_at_3 + value: 96.12403100775194 + - type: mrr_at_5 + value: 96.12403100775194 + - type: nauc_map_at_1000_diff1 + value: -30.073207022158744 + - type: nauc_map_at_1000_max + value: 8.605927047819053 + - type: nauc_map_at_1000_std + value: 37.32903281474912 + - type: nauc_map_at_100_diff1 + value: -10.793039700916037 + - type: nauc_map_at_100_max + value: 5.083492562497569 + - type: nauc_map_at_100_std + value: 8.334344416036425 + - type: nauc_map_at_10_diff1 + value: 8.264796178308766 + - type: nauc_map_at_10_max + value: 4.908570142618324 + - type: nauc_map_at_10_std + value: -18.694986729829253 + - type: nauc_map_at_1_diff1 + value: -2.5494370679913216 + - type: nauc_map_at_1_max + value: -3.9997308413847374 + - type: nauc_map_at_1_std + value: -30.597394453337923 + - type: nauc_map_at_20_diff1 + value: 5.012931007828712 + - type: nauc_map_at_20_max + value: 3.8701830739554888 + - type: nauc_map_at_20_std + value: -12.533557720528027 + - type: nauc_map_at_3_diff1 + value: 5.422969595695591 + - type: nauc_map_at_3_max + value: -4.407058158133122 + - type: nauc_map_at_3_std + value: -27.119262460785613 + - type: nauc_map_at_5_diff1 + value: 5.805119415442102 + - type: nauc_map_at_5_max + value: -1.921645009171383 + - type: nauc_map_at_5_std + value: -24.38322016473787 + - type: nauc_mrr_at_1000_diff1 + value: -29.462717213582252 + - type: nauc_mrr_at_1000_max + value: 52.79303395566356 + - type: nauc_mrr_at_1000_std + value: 82.11480791411556 + - type: nauc_mrr_at_100_diff1 + value: -29.462717213582252 + - type: nauc_mrr_at_100_max + value: 52.79303395566356 + - type: nauc_mrr_at_100_std + value: 82.11480791411556 + - type: nauc_mrr_at_10_diff1 + value: -29.462717213582252 + - type: nauc_mrr_at_10_max + value: 52.79303395566356 + - type: nauc_mrr_at_10_std + value: 82.11480791411556 + - type: nauc_mrr_at_1_diff1 + value: -34.04083805467882 + - type: nauc_mrr_at_1_max + value: 60.66086162971976 + - type: nauc_mrr_at_1_std + value: 85.09567326176314 + - type: nauc_mrr_at_20_diff1 + value: -29.462717213582252 + - type: nauc_mrr_at_20_max + value: 52.79303395566356 + - type: nauc_mrr_at_20_std + value: 82.11480791411556 + - type: nauc_mrr_at_3_diff1 + value: -29.462717213582252 + - type: nauc_mrr_at_3_max + value: 52.79303395566356 + - type: nauc_mrr_at_3_std + value: 82.11480791411556 + - type: nauc_mrr_at_5_diff1 + value: -29.462717213582252 + - type: nauc_mrr_at_5_max + value: 52.79303395566356 + - type: nauc_mrr_at_5_std + value: 82.11480791411556 + - type: nauc_ndcg_at_1000_diff1 + value: -36.970459719408865 + - type: nauc_ndcg_at_1000_max + value: 29.247154299753376 + - type: nauc_ndcg_at_1000_std + value: 47.780059963788766 + - type: nauc_ndcg_at_100_diff1 + value: -30.14872047007895 + - type: nauc_ndcg_at_100_max + value: 14.514360797886198 + - type: nauc_ndcg_at_100_std + value: 43.811084926974644 + - type: nauc_ndcg_at_10_diff1 + value: -18.65882323064364 + - type: nauc_ndcg_at_10_max + value: 19.840743372286695 + - type: nauc_ndcg_at_10_std + value: 24.19536725953071 + - type: nauc_ndcg_at_1_diff1 + value: 2.7525487880013357 + - type: nauc_ndcg_at_1_max + value: 35.518927219904754 + - type: nauc_ndcg_at_1_std + value: 23.38153106486627 + - type: nauc_ndcg_at_20_diff1 + value: -23.495456077508003 + - type: nauc_ndcg_at_20_max + value: 13.690919788099109 + - type: nauc_ndcg_at_20_std + value: 28.561027401967404 + - type: nauc_ndcg_at_3_diff1 + value: 2.692194097409563 + - type: nauc_ndcg_at_3_max + value: 19.104555823234936 + - type: nauc_ndcg_at_3_std + value: 16.108183347865467 + - type: nauc_ndcg_at_5_diff1 + value: -6.012321559676618 + - type: nauc_ndcg_at_5_max + value: 16.82380641991499 + - type: nauc_ndcg_at_5_std + value: 18.37106340133568 + - type: nauc_precision_at_1000_diff1 + value: -27.56580440315084 + - type: nauc_precision_at_1000_max + value: 0.07286293389240653 + - type: nauc_precision_at_1000_std + value: 43.506468401911576 + - type: nauc_precision_at_100_diff1 + value: -32.94712859840603 + - type: nauc_precision_at_100_max + value: 0.6060160329626728 + - type: nauc_precision_at_100_std + value: 43.02595804999937 + - type: nauc_precision_at_10_diff1 + value: -32.017945972534726 + - type: nauc_precision_at_10_max + value: 16.156828736086485 + - type: nauc_precision_at_10_std + value: 46.72951815627366 + - type: nauc_precision_at_1_diff1 + value: -34.04083805467882 + - type: nauc_precision_at_1_max + value: 60.66086162971976 + - type: nauc_precision_at_1_std + value: 85.09567326176314 + - type: nauc_precision_at_20_diff1 + value: -32.86966866791863 + - type: nauc_precision_at_20_max + value: 9.783160758186446 + - type: nauc_precision_at_20_std + value: 47.9737660665705 + - type: nauc_precision_at_3_diff1 + value: -20.08120136563584 + - type: nauc_precision_at_3_max + value: 1.1121036520604461 + - type: nauc_precision_at_3_std + value: 42.92802259504359 + - type: nauc_precision_at_5_diff1 + value: -21.39805327523998 + - type: nauc_precision_at_5_max + value: 9.953438498314199 + - type: nauc_precision_at_5_std + value: 43.21143772394454 + - type: nauc_recall_at_1000_diff1 + value: -46.04578136419592 + - type: nauc_recall_at_1000_max + value: 30.68206598609905 + - type: nauc_recall_at_1000_std + value: 41.050064321026 + - type: nauc_recall_at_100_diff1 + value: -9.355386624701246 + - type: nauc_recall_at_100_max + value: 3.5525078359890814 + - type: nauc_recall_at_100_std + value: -0.14908810395493877 + - type: nauc_recall_at_10_diff1 + value: 6.465716101232745 + - type: nauc_recall_at_10_max + value: 1.5427400921146073 + - type: nauc_recall_at_10_std + value: -24.315672394886402 + - type: nauc_recall_at_1_diff1 + value: -2.5494370679913216 + - type: nauc_recall_at_1_max + value: -3.9997308413847374 + - type: nauc_recall_at_1_std + value: -30.597394453337923 + - type: nauc_recall_at_20_diff1 + value: 5.297515311292863 + - type: nauc_recall_at_20_max + value: 1.013910149606766 + - type: nauc_recall_at_20_std + value: -18.419460527980615 + - type: nauc_recall_at_3_diff1 + value: 3.8520264716457433 + - type: nauc_recall_at_3_max + value: -6.683530905079106 + - type: nauc_recall_at_3_std + value: -31.104191625229692 + - type: nauc_recall_at_5_diff1 + value: 4.0202738409290975 + - type: nauc_recall_at_5_max + value: -3.4826228221657556 + - type: nauc_recall_at_5_std + value: -28.291427757615367 + - type: ndcg_at_1 + value: 73.643 + - type: ndcg_at_10 + value: 74.044 + - type: ndcg_at_100 + value: 68.757 + - type: ndcg_at_1000 + value: 77.934 + - type: ndcg_at_20 + value: 73.043 + - type: ndcg_at_3 + value: 74.994 + - type: ndcg_at_5 + value: 73.014 + - type: precision_at_1 + value: 93.023 + - type: precision_at_10 + value: 84.186 + - type: precision_at_100 + value: 41.186 + - type: precision_at_1000 + value: 8.116 + - type: precision_at_20 + value: 77.442 + - type: precision_at_3 + value: 89.922 + - type: precision_at_5 + value: 85.116 + - type: recall_at_1 + value: 2.307 + - type: recall_at_10 + value: 18.497 + - type: recall_at_100 + value: 56.711999999999996 + - type: recall_at_1000 + value: 86.737 + - type: recall_at_20 + value: 30.532999999999998 + - type: recall_at_3 + value: 6.4399999999999995 + - type: recall_at_5 + value: 10.105 + task: + type: Retrieval + - dataset: + config: en + name: MTEB MTOPDomainClassification (en) + revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf + split: test + type: mteb/mtop_domain + metrics: + - type: accuracy + value: 96.35430916552667 + - type: f1 + value: 96.16669219074517 + - type: f1_weighted + value: 96.35506582065435 + - type: main_score + value: 96.35430916552667 + task: + type: Classification + - dataset: + config: en + name: MTEB MTOPIntentClassification (en) + revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba + split: test + type: mteb/mtop_intent + metrics: + - type: accuracy + value: 80.51527587779297 + - type: f1 + value: 59.350461259612345 + - type: f1_weighted + value: 81.51891267687044 + - type: main_score + value: 80.51527587779297 + task: + type: Classification + - dataset: + config: en + name: MTEB MassiveIntentClassification (en) + revision: 4672e20407010da34463acc759c162ca9734bca6 + split: test + type: mteb/amazon_massive_intent + metrics: + - type: accuracy + value: 77.31338264963013 + - type: f1 + value: 75.29547524788576 + - type: f1_weighted + value: 76.26831259224058 + - type: main_score + value: 77.31338264963013 + task: + type: Classification + - dataset: + config: en + name: MTEB MassiveScenarioClassification (en) + revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 + split: test + type: mteb/amazon_massive_scenario + metrics: + - type: accuracy + value: 79.97982515131137 + - type: f1 + value: 79.34057805450769 + - type: f1_weighted + value: 79.73023446597212 + - type: main_score + value: 79.97982515131137 + task: + type: Classification + - dataset: + config: default + name: MTEB MedrxivClusteringP2P (default) + revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 + split: test + type: mteb/medrxiv-clustering-p2p + metrics: + - type: main_score + value: 38.37635785818304 + - type: v_measure + value: 38.37635785818304 + - type: v_measure_std + value: 1.6943794496059137 + task: + type: Clustering + - dataset: + config: default + name: MTEB MedrxivClusteringS2S (default) + revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 + split: test + type: mteb/medrxiv-clustering-s2s + metrics: + - type: main_score + value: 37.6711034083755 + - type: v_measure + value: 37.6711034083755 + - type: v_measure_std + value: 1.1408887612104992 + task: + type: Clustering + - dataset: + config: default + name: MTEB MindSmallReranking (default) + revision: 59042f120c80e8afa9cdbb224f67076cec0fc9a7 + split: test + type: mteb/mind_small + metrics: + - type: main_score + value: 32.20170969306457 + - type: map + value: 32.20170969306457 + - type: mrr + value: 33.41738896071552 + - type: nAUC_map_diff1 + value: 12.077124363492512 + - type: nAUC_map_max + value: -20.336429990396454 + - type: nAUC_map_std + value: 0.10724031251638018 + - type: nAUC_mrr_diff1 + value: 11.405695518900744 + - type: nAUC_mrr_max + value: -15.0727490448132 + - type: nAUC_mrr_std + value: 1.8987958512727106 + task: + type: Reranking + - dataset: + config: default + name: MTEB NFCorpus (default) + revision: ec0fa4fe99da2ff19ca1214b7966684033a58814 + split: test + type: mteb/nfcorpus + metrics: + - type: main_score + value: 40.463 + - type: map_at_1 + value: 6.4990000000000006 + - type: map_at_10 + value: 15.699 + - type: map_at_100 + value: 19.895 + - type: map_at_1000 + value: 21.537 + - type: map_at_20 + value: 17.429 + - type: map_at_3 + value: 11.48 + - type: map_at_5 + value: 13.383999999999999 + - type: mrr_at_1 + value: 52.63157894736842 + - type: mrr_at_10 + value: 61.60401002506265 + - type: mrr_at_100 + value: 62.04336653809741 + - type: mrr_at_1000 + value: 62.07610833363911 + - type: mrr_at_20 + value: 61.88033067968176 + - type: mrr_at_3 + value: 59.44272445820435 + - type: mrr_at_5 + value: 60.89783281733746 + - type: nauc_map_at_1000_diff1 + value: 18.58585974547791 + - type: nauc_map_at_1000_max + value: 30.25465935470905 + - type: nauc_map_at_1000_std + value: 10.987080017051682 + - type: nauc_map_at_100_diff1 + value: 20.02651798573329 + - type: nauc_map_at_100_max + value: 30.108719787095467 + - type: nauc_map_at_100_std + value: 7.882019722247158 + - type: nauc_map_at_10_diff1 + value: 23.02800157136177 + - type: nauc_map_at_10_max + value: 22.8723397741279 + - type: nauc_map_at_10_std + value: -3.762893117006399 + - type: nauc_map_at_1_diff1 + value: 37.94611136294878 + - type: nauc_map_at_1_max + value: 7.297492349938244 + - type: nauc_map_at_1_std + value: -17.813930346562152 + - type: nauc_map_at_20_diff1 + value: 21.981440837881113 + - type: nauc_map_at_20_max + value: 26.759497880383837 + - type: nauc_map_at_20_std + value: 0.18040330674839283 + - type: nauc_map_at_3_diff1 + value: 27.066009968256555 + - type: nauc_map_at_3_max + value: 10.488797596450187 + - type: nauc_map_at_3_std + value: -14.013059830876845 + - type: nauc_map_at_5_diff1 + value: 25.493785001708446 + - type: nauc_map_at_5_max + value: 16.217756878539337 + - type: nauc_map_at_5_std + value: -10.714238788014212 + - type: nauc_mrr_at_1000_diff1 + value: 28.488264933723528 + - type: nauc_mrr_at_1000_max + value: 45.94151165403325 + - type: nauc_mrr_at_1000_std + value: 25.20231778025588 + - type: nauc_mrr_at_100_diff1 + value: 28.4886630218298 + - type: nauc_mrr_at_100_max + value: 45.9702575916014 + - type: nauc_mrr_at_100_std + value: 25.22848732842774 + - type: nauc_mrr_at_10_diff1 + value: 28.535257017998294 + - type: nauc_mrr_at_10_max + value: 45.86005605851268 + - type: nauc_mrr_at_10_std + value: 24.81744203643852 + - type: nauc_mrr_at_1_diff1 + value: 29.824630548327285 + - type: nauc_mrr_at_1_max + value: 44.19891968145314 + - type: nauc_mrr_at_1_std + value: 23.21413139777098 + - type: nauc_mrr_at_20_diff1 + value: 28.54642005356483 + - type: nauc_mrr_at_20_max + value: 46.08926361963997 + - type: nauc_mrr_at_20_std + value: 25.39517294920476 + - type: nauc_mrr_at_3_diff1 + value: 28.230929109259407 + - type: nauc_mrr_at_3_max + value: 44.05364599618201 + - type: nauc_mrr_at_3_std + value: 23.828100697992724 + - type: nauc_mrr_at_5_diff1 + value: 29.669751924690758 + - type: nauc_mrr_at_5_max + value: 45.36862661497384 + - type: nauc_mrr_at_5_std + value: 23.787166807022505 + - type: nauc_ndcg_at_1000_diff1 + value: 18.515898773404377 + - type: nauc_ndcg_at_1000_max + value: 44.57748675979855 + - type: nauc_ndcg_at_1000_std + value: 29.205899131269604 + - type: nauc_ndcg_at_100_diff1 + value: 15.88197701276405 + - type: nauc_ndcg_at_100_max + value: 39.62665883972109 + - type: nauc_ndcg_at_100_std + value: 25.186347352251754 + - type: nauc_ndcg_at_10_diff1 + value: 16.220798038950925 + - type: nauc_ndcg_at_10_max + value: 39.67757337154769 + - type: nauc_ndcg_at_10_std + value: 25.634534917262403 + - type: nauc_ndcg_at_1_diff1 + value: 31.448775879462932 + - type: nauc_ndcg_at_1_max + value: 44.4256421079556 + - type: nauc_ndcg_at_1_std + value: 23.093987850437355 + - type: nauc_ndcg_at_20_diff1 + value: 15.417507391228035 + - type: nauc_ndcg_at_20_max + value: 37.52014353976055 + - type: nauc_ndcg_at_20_std + value: 23.880617920537915 + - type: nauc_ndcg_at_3_diff1 + value: 18.01018470616153 + - type: nauc_ndcg_at_3_max + value: 39.135814950810804 + - type: nauc_ndcg_at_3_std + value: 21.40850285781106 + - type: nauc_ndcg_at_5_diff1 + value: 18.502338826072368 + - type: nauc_ndcg_at_5_max + value: 40.2043937728194 + - type: nauc_ndcg_at_5_std + value: 22.242499743433424 + - type: nauc_precision_at_1000_diff1 + value: -13.648652068964681 + - type: nauc_precision_at_1000_max + value: 3.5821865423513426 + - type: nauc_precision_at_1000_std + value: 35.481456041211274 + - type: nauc_precision_at_100_diff1 + value: -11.342790040792961 + - type: nauc_precision_at_100_max + value: 18.41811151847882 + - type: nauc_precision_at_100_std + value: 44.901842372597336 + - type: nauc_precision_at_10_diff1 + value: -1.9404654865248405 + - type: nauc_precision_at_10_max + value: 40.91955602631143 + - type: nauc_precision_at_10_std + value: 41.38128398646734 + - type: nauc_precision_at_1_diff1 + value: 29.824630548327285 + - type: nauc_precision_at_1_max + value: 44.19891968145314 + - type: nauc_precision_at_1_std + value: 23.21413139777098 + - type: nauc_precision_at_20_diff1 + value: -5.046696327994225 + - type: nauc_precision_at_20_max + value: 33.653422186725386 + - type: nauc_precision_at_20_std + value: 40.97689615511939 + - type: nauc_precision_at_3_diff1 + value: 5.1767717826900785 + - type: nauc_precision_at_3_max + value: 38.01276130261592 + - type: nauc_precision_at_3_std + value: 25.71468883159735 + - type: nauc_precision_at_5_diff1 + value: 3.847065262189492 + - type: nauc_precision_at_5_max + value: 41.00941977122254 + - type: nauc_precision_at_5_std + value: 31.044768384177246 + - type: nauc_recall_at_1000_diff1 + value: 7.975632504947066 + - type: nauc_recall_at_1000_max + value: 18.83264064904865 + - type: nauc_recall_at_1000_std + value: 15.023940189337717 + - type: nauc_recall_at_100_diff1 + value: 10.354458867884487 + - type: nauc_recall_at_100_max + value: 27.16900376430975 + - type: nauc_recall_at_100_std + value: 14.160333284050214 + - type: nauc_recall_at_10_diff1 + value: 18.04347857307359 + - type: nauc_recall_at_10_max + value: 19.082544744457774 + - type: nauc_recall_at_10_std + value: -5.107813434157397 + - type: nauc_recall_at_1_diff1 + value: 37.94611136294878 + - type: nauc_recall_at_1_max + value: 7.297492349938244 + - type: nauc_recall_at_1_std + value: -17.813930346562152 + - type: nauc_recall_at_20_diff1 + value: 16.658153504941193 + - type: nauc_recall_at_20_max + value: 23.214261213582382 + - type: nauc_recall_at_20_std + value: -0.6964816170313349 + - type: nauc_recall_at_3_diff1 + value: 23.65600569767465 + - type: nauc_recall_at_3_max + value: 6.543906048065431 + - type: nauc_recall_at_3_std + value: -15.496093666790777 + - type: nauc_recall_at_5_diff1 + value: 22.112315726267077 + - type: nauc_recall_at_5_max + value: 12.258969896916307 + - type: nauc_recall_at_5_std + value: -12.922832334587008 + - type: ndcg_at_1 + value: 50.929 + - type: ndcg_at_10 + value: 40.463 + - type: ndcg_at_100 + value: 36.909 + - type: ndcg_at_1000 + value: 45.617999999999995 + - type: ndcg_at_20 + value: 37.772 + - type: ndcg_at_3 + value: 46.315 + - type: ndcg_at_5 + value: 44.052 + - type: precision_at_1 + value: 52.632 + - type: precision_at_10 + value: 29.814 + - type: precision_at_100 + value: 9.325 + - type: precision_at_1000 + value: 2.236 + - type: precision_at_20 + value: 22.073999999999998 + - type: precision_at_3 + value: 42.931000000000004 + - type: precision_at_5 + value: 37.957 + - type: recall_at_1 + value: 6.4990000000000006 + - type: recall_at_10 + value: 20.232 + - type: recall_at_100 + value: 36.846000000000004 + - type: recall_at_1000 + value: 69.03 + - type: recall_at_20 + value: 24.448 + - type: recall_at_3 + value: 13.258000000000001 + - type: recall_at_5 + value: 16.255 + task: + type: Retrieval + - dataset: + config: default + name: MTEB NQ (default) + revision: b774495ed302d8c44a3a7ea25c90dbce03968f31 + split: test + type: mteb/nq + metrics: + - type: main_score + value: 70.518 + - type: map_at_1 + value: 46.233999999999995 + - type: map_at_10 + value: 63.519999999999996 + - type: map_at_100 + value: 64.14699999999999 + - type: map_at_1000 + value: 64.154 + - type: map_at_20 + value: 63.975 + - type: map_at_3 + value: 59.797 + - type: map_at_5 + value: 62.226000000000006 + - type: mrr_at_1 + value: 51.76709154113557 + - type: mrr_at_10 + value: 65.79852489470095 + - type: mrr_at_100 + value: 66.19480681115492 + - type: mrr_at_1000 + value: 66.19993656063721 + - type: mrr_at_20 + value: 66.0923632685851 + - type: mrr_at_3 + value: 63.185592893008746 + - type: mrr_at_5 + value: 64.93385477018151 + - type: nauc_map_at_1000_diff1 + value: 43.35077155361084 + - type: nauc_map_at_1000_max + value: 37.282536180921085 + - type: nauc_map_at_1000_std + value: -4.64357984773174 + - type: nauc_map_at_100_diff1 + value: 43.35098576601616 + - type: nauc_map_at_100_max + value: 37.28998747522813 + - type: nauc_map_at_100_std + value: -4.638151362399621 + - type: nauc_map_at_10_diff1 + value: 43.131007214082594 + - type: nauc_map_at_10_max + value: 37.430076712266846 + - type: nauc_map_at_10_std + value: -4.90614475410035 + - type: nauc_map_at_1_diff1 + value: 45.843123692592485 + - type: nauc_map_at_1_max + value: 30.160164681399227 + - type: nauc_map_at_1_std + value: -6.110582951655118 + - type: nauc_map_at_20_diff1 + value: 43.30588135441681 + - type: nauc_map_at_20_max + value: 37.41321766111187 + - type: nauc_map_at_20_std + value: -4.628074353861448 + - type: nauc_map_at_3_diff1 + value: 42.690411835598695 + - type: nauc_map_at_3_max + value: 36.64069333510947 + - type: nauc_map_at_3_std + value: -6.2899609993355545 + - type: nauc_map_at_5_diff1 + value: 42.906814471744134 + - type: nauc_map_at_5_max + value: 37.27599132551781 + - type: nauc_map_at_5_std + value: -5.512203849661435 + - type: nauc_mrr_at_1000_diff1 + value: 43.77989113830799 + - type: nauc_mrr_at_1000_max + value: 38.01009876981156 + - type: nauc_mrr_at_1000_std + value: -2.0250764367321654 + - type: nauc_mrr_at_100_diff1 + value: 43.78071481914773 + - type: nauc_mrr_at_100_max + value: 38.01603112272088 + - type: nauc_mrr_at_100_std + value: -2.019685020907906 + - type: nauc_mrr_at_10_diff1 + value: 43.582338882429156 + - type: nauc_mrr_at_10_max + value: 38.19577506300954 + - type: nauc_mrr_at_10_std + value: -2.011905402842086 + - type: nauc_mrr_at_1_diff1 + value: 46.544635554669576 + - type: nauc_mrr_at_1_max + value: 33.82720628969995 + - type: nauc_mrr_at_1_std + value: -2.924293824382781 + - type: nauc_mrr_at_20_diff1 + value: 43.713682995581614 + - type: nauc_mrr_at_20_max + value: 38.09918392374771 + - type: nauc_mrr_at_20_std + value: -1.9583477023239 + - type: nauc_mrr_at_3_diff1 + value: 43.35807398052401 + - type: nauc_mrr_at_3_max + value: 38.39129780935902 + - type: nauc_mrr_at_3_std + value: -2.287791352096624 + - type: nauc_mrr_at_5_diff1 + value: 43.4126448419642 + - type: nauc_mrr_at_5_max + value: 38.27294037073721 + - type: nauc_mrr_at_5_std + value: -2.166655666337289 + - type: nauc_ndcg_at_1000_diff1 + value: 43.26202839737687 + - type: nauc_ndcg_at_1000_max + value: 38.493273787010615 + - type: nauc_ndcg_at_1000_std + value: -2.9983001465713524 + - type: nauc_ndcg_at_100_diff1 + value: 43.25688556190981 + - type: nauc_ndcg_at_100_max + value: 38.68155788574137 + - type: nauc_ndcg_at_100_std + value: -2.8355616191757487 + - type: nauc_ndcg_at_10_diff1 + value: 42.37071983774907 + - type: nauc_ndcg_at_10_max + value: 39.60970139451164 + - type: nauc_ndcg_at_10_std + value: -3.5877671856177775 + - type: nauc_ndcg_at_1_diff1 + value: 46.614780156517845 + - type: nauc_ndcg_at_1_max + value: 33.863655999315526 + - type: nauc_ndcg_at_1_std + value: -2.839239881422542 + - type: nauc_ndcg_at_20_diff1 + value: 42.97845395193472 + - type: nauc_ndcg_at_20_max + value: 39.53053589334249 + - type: nauc_ndcg_at_20_std + value: -2.6507495263904515 + - type: nauc_ndcg_at_3_diff1 + value: 41.65390869521735 + - type: nauc_ndcg_at_3_max + value: 38.4851846089685 + - type: nauc_ndcg_at_3_std + value: -5.6296606018146 + - type: nauc_ndcg_at_5_diff1 + value: 41.89640848285409 + - type: nauc_ndcg_at_5_max + value: 39.293659812249615 + - type: nauc_ndcg_at_5_std + value: -4.754462409312945 + - type: nauc_precision_at_1000_diff1 + value: -10.848480634403051 + - type: nauc_precision_at_1000_max + value: 1.3436973699935175 + - type: nauc_precision_at_1000_std + value: 19.044141500097957 + - type: nauc_precision_at_100_diff1 + value: -9.018095533261604 + - type: nauc_precision_at_100_max + value: 4.0402155161025695 + - type: nauc_precision_at_100_std + value: 19.492823636364996 + - type: nauc_precision_at_10_diff1 + value: 3.947100636096294 + - type: nauc_precision_at_10_max + value: 20.598641503195907 + - type: nauc_precision_at_10_std + value: 13.522240087840858 + - type: nauc_precision_at_1_diff1 + value: 46.614780156517845 + - type: nauc_precision_at_1_max + value: 33.863655999315526 + - type: nauc_precision_at_1_std + value: -2.839239881422542 + - type: nauc_precision_at_20_diff1 + value: -2.1791072352475336 + - type: nauc_precision_at_20_max + value: 14.03887841842901 + - type: nauc_precision_at_20_std + value: 18.846129471001632 + - type: nauc_precision_at_3_diff1 + value: 21.09092861833543 + - type: nauc_precision_at_3_max + value: 34.122841034361805 + - type: nauc_precision_at_3_std + value: 2.5513201020031064 + - type: nauc_precision_at_5_diff1 + value: 12.181140062410874 + - type: nauc_precision_at_5_max + value: 27.903435474574234 + - type: nauc_precision_at_5_std + value: 7.6589638998570315 + - type: nauc_recall_at_1000_diff1 + value: 59.28482230176634 + - type: nauc_recall_at_1000_max + value: 85.47306385133284 + - type: nauc_recall_at_1000_std + value: 76.45740117805659 + - type: nauc_recall_at_100_diff1 + value: 44.31190730138568 + - type: nauc_recall_at_100_max + value: 66.30976579719086 + - type: nauc_recall_at_100_std + value: 30.65274229759539 + - type: nauc_recall_at_10_diff1 + value: 34.885747244334866 + - type: nauc_recall_at_10_max + value: 50.998198327439404 + - type: nauc_recall_at_10_std + value: -2.7025509359838193 + - type: nauc_recall_at_1_diff1 + value: 45.843123692592485 + - type: nauc_recall_at_1_max + value: 30.160164681399227 + - type: nauc_recall_at_1_std + value: -6.110582951655118 + - type: nauc_recall_at_20_diff1 + value: 37.873054394800825 + - type: nauc_recall_at_20_max + value: 59.21039923637266 + - type: nauc_recall_at_20_std + value: 9.352312696050557 + - type: nauc_recall_at_3_diff1 + value: 35.703271085627776 + - type: nauc_recall_at_3_max + value: 41.19400688280121 + - type: nauc_recall_at_3_std + value: -7.9624895195139 + - type: nauc_recall_at_5_diff1 + value: 34.831972383157925 + - type: nauc_recall_at_5_max + value: 44.82018386701478 + - type: nauc_recall_at_5_std + value: -7.046789506164082 + - type: ndcg_at_1 + value: 51.73799999999999 + - type: ndcg_at_10 + value: 70.518 + - type: ndcg_at_100 + value: 72.841 + - type: ndcg_at_1000 + value: 72.99799999999999 + - type: ndcg_at_20 + value: 71.895 + - type: ndcg_at_3 + value: 64.06500000000001 + - type: ndcg_at_5 + value: 67.86999999999999 + - type: precision_at_1 + value: 51.73799999999999 + - type: precision_at_10 + value: 10.698 + - type: precision_at_100 + value: 1.2 + - type: precision_at_1000 + value: 0.121 + - type: precision_at_20 + value: 5.691 + - type: precision_at_3 + value: 28.37 + - type: precision_at_5 + value: 19.363 + - type: recall_at_1 + value: 46.233999999999995 + - type: recall_at_10 + value: 89.062 + - type: recall_at_100 + value: 98.622 + - type: recall_at_1000 + value: 99.754 + - type: recall_at_20 + value: 94.052 + - type: recall_at_3 + value: 72.994 + - type: recall_at_5 + value: 81.525 + task: + type: Retrieval + - dataset: + config: default + name: MTEB QuoraRetrieval (default) + revision: e4e08e0b7dbe3c8700f0daef558ff32256715259 + split: test + type: mteb/quora + metrics: + - type: main_score + value: 89.885 + - type: map_at_1 + value: 72.379 + - type: map_at_10 + value: 86.455 + - type: map_at_100 + value: 87.087 + - type: map_at_1000 + value: 87.1 + - type: map_at_20 + value: 86.883 + - type: map_at_3 + value: 83.663 + - type: map_at_5 + value: 85.443 + - type: mrr_at_1 + value: 83.27 + - type: mrr_at_10 + value: 89.13586904761888 + - type: mrr_at_100 + value: 89.22177886254626 + - type: mrr_at_1000 + value: 89.22204575963424 + - type: mrr_at_20 + value: 89.20621913458041 + - type: mrr_at_3 + value: 88.37999999999981 + - type: mrr_at_5 + value: 88.89349999999978 + - type: nauc_map_at_1000_diff1 + value: 78.17410401832315 + - type: nauc_map_at_1000_max + value: 33.114749237960986 + - type: nauc_map_at_1000_std + value: -49.79724283243796 + - type: nauc_map_at_100_diff1 + value: 78.17873434671671 + - type: nauc_map_at_100_max + value: 33.101626543573325 + - type: nauc_map_at_100_std + value: -49.82883017160494 + - type: nauc_map_at_10_diff1 + value: 78.28052682172311 + - type: nauc_map_at_10_max + value: 32.626693803188694 + - type: nauc_map_at_10_std + value: -51.941057676350034 + - type: nauc_map_at_1_diff1 + value: 81.06079816824507 + - type: nauc_map_at_1_max + value: 25.638093235123616 + - type: nauc_map_at_1_std + value: -43.230210939240344 + - type: nauc_map_at_20_diff1 + value: 78.22103944842512 + - type: nauc_map_at_20_max + value: 32.94488423505404 + - type: nauc_map_at_20_std + value: -50.69181407781227 + - type: nauc_map_at_3_diff1 + value: 78.75453877967588 + - type: nauc_map_at_3_max + value: 30.645950847243686 + - type: nauc_map_at_3_std + value: -52.983886453956266 + - type: nauc_map_at_5_diff1 + value: 78.44984884302167 + - type: nauc_map_at_5_max + value: 31.69697839442234 + - type: nauc_map_at_5_std + value: -53.21480554718401 + - type: nauc_mrr_at_1000_diff1 + value: 78.90502271071976 + - type: nauc_mrr_at_1000_max + value: 35.902725888631075 + - type: nauc_mrr_at_1000_std + value: -45.82579843551156 + - type: nauc_mrr_at_100_diff1 + value: 78.90552803580407 + - type: nauc_mrr_at_100_max + value: 35.90392790964254 + - type: nauc_mrr_at_100_std + value: -45.82489205475015 + - type: nauc_mrr_at_10_diff1 + value: 78.89432223469271 + - type: nauc_mrr_at_10_max + value: 35.86669566861425 + - type: nauc_mrr_at_10_std + value: -46.0616841694464 + - type: nauc_mrr_at_1_diff1 + value: 79.53513360034344 + - type: nauc_mrr_at_1_max + value: 35.299514657188006 + - type: nauc_mrr_at_1_std + value: -43.17936948437256 + - type: nauc_mrr_at_20_diff1 + value: 78.90707352031835 + - type: nauc_mrr_at_20_max + value: 35.906499072241296 + - type: nauc_mrr_at_20_std + value: -45.8904084451193 + - type: nauc_mrr_at_3_diff1 + value: 78.70913062166218 + - type: nauc_mrr_at_3_max + value: 36.16709621132144 + - type: nauc_mrr_at_3_std + value: -46.00948004774822 + - type: nauc_mrr_at_5_diff1 + value: 78.91095031555673 + - type: nauc_mrr_at_5_max + value: 36.010878683954566 + - type: nauc_mrr_at_5_std + value: -46.31731368609175 + - type: nauc_ndcg_at_1000_diff1 + value: 78.19132492477127 + - type: nauc_ndcg_at_1000_max + value: 34.5208358892501 + - type: nauc_ndcg_at_1000_std + value: -47.938360906488974 + - type: nauc_ndcg_at_100_diff1 + value: 78.24549799575261 + - type: nauc_ndcg_at_100_max + value: 34.48869025578818 + - type: nauc_ndcg_at_100_std + value: -48.02996375451253 + - type: nauc_ndcg_at_10_diff1 + value: 78.15340584208084 + - type: nauc_ndcg_at_10_max + value: 33.5226981818058 + - type: nauc_ndcg_at_10_std + value: -51.690477519601494 + - type: nauc_ndcg_at_1_diff1 + value: 79.55459365767561 + - type: nauc_ndcg_at_1_max + value: 35.25214101433387 + - type: nauc_ndcg_at_1_std + value: -43.10088819860409 + - type: nauc_ndcg_at_20_diff1 + value: 78.27277286768546 + - type: nauc_ndcg_at_20_max + value: 33.997104745595564 + - type: nauc_ndcg_at_20_std + value: -50.10549601980995 + - type: nauc_ndcg_at_3_diff1 + value: 77.68820501917479 + - type: nauc_ndcg_at_3_max + value: 33.00389630941839 + - type: nauc_ndcg_at_3_std + value: -51.00595251236665 + - type: nauc_ndcg_at_5_diff1 + value: 78.08093149961476 + - type: nauc_ndcg_at_5_max + value: 33.03434664578743 + - type: nauc_ndcg_at_5_std + value: -52.37122386447497 + - type: nauc_precision_at_1000_diff1 + value: -44.49830608740945 + - type: nauc_precision_at_1000_max + value: -7.3283280714307395 + - type: nauc_precision_at_1000_std + value: 38.55076692876393 + - type: nauc_precision_at_100_diff1 + value: -44.252675314263904 + - type: nauc_precision_at_100_max + value: -7.038454433556829 + - type: nauc_precision_at_100_std + value: 38.247323997481615 + - type: nauc_precision_at_10_diff1 + value: -40.192852013615216 + - type: nauc_precision_at_10_max + value: -3.7258976649568036 + - type: nauc_precision_at_10_std + value: 25.983458444206182 + - type: nauc_precision_at_1_diff1 + value: 79.55459365767561 + - type: nauc_precision_at_1_max + value: 35.25214101433387 + - type: nauc_precision_at_1_std + value: -43.10088819860409 + - type: nauc_precision_at_20_diff1 + value: -43.020749754821495 + - type: nauc_precision_at_20_max + value: -5.7062060443801075 + - type: nauc_precision_at_20_std + value: 32.8862431943092 + - type: nauc_precision_at_3_diff1 + value: -22.843593386293996 + - type: nauc_precision_at_3_max + value: 4.474275296763041 + - type: nauc_precision_at_3_std + value: 6.119920479600398 + - type: nauc_precision_at_5_diff1 + value: -33.598088334605045 + - type: nauc_precision_at_5_max + value: -0.41505757559350775 + - type: nauc_precision_at_5_std + value: 16.52526817965026 + - type: nauc_recall_at_1000_diff1 + value: 28.726073762912847 + - type: nauc_recall_at_1000_max + value: -57.390873015654066 + - type: nauc_recall_at_1000_std + value: 69.71288515421948 + - type: nauc_recall_at_100_diff1 + value: 83.06070133460443 + - type: nauc_recall_at_100_max + value: 33.27991294763942 + - type: nauc_recall_at_100_std + value: -42.785112479889655 + - type: nauc_recall_at_10_diff1 + value: 74.73877865072825 + - type: nauc_recall_at_10_max + value: 27.81410621945221 + - type: nauc_recall_at_10_std + value: -75.85371099008806 + - type: nauc_recall_at_1_diff1 + value: 81.06079816824507 + - type: nauc_recall_at_1_max + value: 25.638093235123616 + - type: nauc_recall_at_1_std + value: -43.230210939240344 + - type: nauc_recall_at_20_diff1 + value: 76.04615040930837 + - type: nauc_recall_at_20_max + value: 27.47173316749929 + - type: nauc_recall_at_20_std + value: -78.29029550423172 + - type: nauc_recall_at_3_diff1 + value: 75.29987903678384 + - type: nauc_recall_at_3_max + value: 27.48543826795177 + - type: nauc_recall_at_3_std + value: -60.91023011356427 + - type: nauc_recall_at_5_diff1 + value: 74.71682412813378 + - type: nauc_recall_at_5_max + value: 27.255092143441562 + - type: nauc_recall_at_5_std + value: -69.03177732393821 + - type: ndcg_at_1 + value: 83.26 + - type: ndcg_at_10 + value: 89.885 + - type: ndcg_at_100 + value: 90.968 + - type: ndcg_at_1000 + value: 91.02799999999999 + - type: ndcg_at_20 + value: 90.52900000000001 + - type: ndcg_at_3 + value: 87.443 + - type: ndcg_at_5 + value: 88.81 + - type: precision_at_1 + value: 83.26 + - type: precision_at_10 + value: 13.581999999999999 + - type: precision_at_100 + value: 1.541 + - type: precision_at_1000 + value: 0.157 + - type: precision_at_20 + value: 7.210999999999999 + - type: precision_at_3 + value: 38.323 + - type: precision_at_5 + value: 25.069999999999997 + - type: recall_at_1 + value: 72.379 + - type: recall_at_10 + value: 96.261 + - type: recall_at_100 + value: 99.779 + - type: recall_at_1000 + value: 99.996 + - type: recall_at_20 + value: 98.301 + - type: recall_at_3 + value: 89.101 + - type: recall_at_5 + value: 93.11500000000001 + task: + type: Retrieval + - dataset: + config: default + name: MTEB RedditClustering (default) + revision: 24640382cdbf8abc73003fb0fa6d111a705499eb + split: test + type: mteb/reddit-clustering + metrics: + - type: main_score + value: 61.87769077476204 + - type: v_measure + value: 61.87769077476204 + - type: v_measure_std + value: 5.290405218730049 + task: + type: Clustering + - dataset: + config: default + name: MTEB RedditClusteringP2P (default) + revision: 385e3cb46b4cfa89021f56c4380204149d0efe33 + split: test + type: mteb/reddit-clustering-p2p + metrics: + - type: main_score + value: 68.29553057563754 + - type: v_measure + value: 68.29553057563754 + - type: v_measure_std + value: 13.019229253711732 + task: + type: Clustering + - dataset: + config: default + name: MTEB SCIDOCS (default) + revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88 + split: test + type: mteb/scidocs + metrics: + - type: main_score + value: 20.86 + - type: map_at_1 + value: 4.843 + - type: map_at_10 + value: 12.457 + - type: map_at_100 + value: 14.648 + - type: map_at_1000 + value: 14.965 + - type: map_at_20 + value: 13.596 + - type: map_at_3 + value: 8.776 + - type: map_at_5 + value: 10.528 + - type: mrr_at_1 + value: 23.799999999999997 + - type: mrr_at_10 + value: 34.93765873015872 + - type: mrr_at_100 + value: 36.054095036751825 + - type: mrr_at_1000 + value: 36.10871797082569 + - type: mrr_at_20 + value: 35.57880859465608 + - type: mrr_at_3 + value: 31.54999999999999 + - type: mrr_at_5 + value: 33.53999999999998 + - type: nauc_map_at_1000_diff1 + value: 16.889540490911525 + - type: nauc_map_at_1000_max + value: 25.726340275186143 + - type: nauc_map_at_1000_std + value: 9.926911665196988 + - type: nauc_map_at_100_diff1 + value: 16.889355521248202 + - type: nauc_map_at_100_max + value: 25.628741550328126 + - type: nauc_map_at_100_std + value: 9.637062917997012 + - type: nauc_map_at_10_diff1 + value: 16.972521218507854 + - type: nauc_map_at_10_max + value: 24.810172126870363 + - type: nauc_map_at_10_std + value: 7.09295422867669 + - type: nauc_map_at_1_diff1 + value: 24.9292922418417 + - type: nauc_map_at_1_max + value: 15.49253311874767 + - type: nauc_map_at_1_std + value: -0.4754734108717385 + - type: nauc_map_at_20_diff1 + value: 16.945564955989113 + - type: nauc_map_at_20_max + value: 25.197327599885362 + - type: nauc_map_at_20_std + value: 7.972256233219635 + - type: nauc_map_at_3_diff1 + value: 19.503723922705067 + - type: nauc_map_at_3_max + value: 20.795879090480057 + - type: nauc_map_at_3_std + value: 1.5828913591118658 + - type: nauc_map_at_5_diff1 + value: 19.80474780705204 + - type: nauc_map_at_5_max + value: 24.040173591299723 + - type: nauc_map_at_5_std + value: 4.153642430396917 + - type: nauc_mrr_at_1000_diff1 + value: 21.80300741603344 + - type: nauc_mrr_at_1000_max + value: 19.98123409846586 + - type: nauc_mrr_at_1000_std + value: 3.6325335777371377 + - type: nauc_mrr_at_100_diff1 + value: 21.804966803578946 + - type: nauc_mrr_at_100_max + value: 19.9965104601956 + - type: nauc_mrr_at_100_std + value: 3.6713772865070107 + - type: nauc_mrr_at_10_diff1 + value: 21.66109150475663 + - type: nauc_mrr_at_10_max + value: 19.873876575424404 + - type: nauc_mrr_at_10_std + value: 3.3387503298795584 + - type: nauc_mrr_at_1_diff1 + value: 24.868548821073084 + - type: nauc_mrr_at_1_max + value: 16.189915011439044 + - type: nauc_mrr_at_1_std + value: -0.17692171251799987 + - type: nauc_mrr_at_20_diff1 + value: 21.677427533247375 + - type: nauc_mrr_at_20_max + value: 19.967193157614872 + - type: nauc_mrr_at_20_std + value: 3.639825799332009 + - type: nauc_mrr_at_3_diff1 + value: 21.681117207511825 + - type: nauc_mrr_at_3_max + value: 19.132660363303295 + - type: nauc_mrr_at_3_std + value: 1.6613642176263752 + - type: nauc_mrr_at_5_diff1 + value: 21.833332207271884 + - type: nauc_mrr_at_5_max + value: 19.926480855266213 + - type: nauc_mrr_at_5_std + value: 2.901801717093585 + - type: nauc_ndcg_at_1000_diff1 + value: 16.92599483752314 + - type: nauc_ndcg_at_1000_max + value: 27.126582080942814 + - type: nauc_ndcg_at_1000_std + value: 16.638448489514683 + - type: nauc_ndcg_at_100_diff1 + value: 16.96586885959473 + - type: nauc_ndcg_at_100_max + value: 26.675878724175046 + - type: nauc_ndcg_at_100_std + value: 15.369335585614245 + - type: nauc_ndcg_at_10_diff1 + value: 16.59779893225997 + - type: nauc_ndcg_at_10_max + value: 24.865338966132818 + - type: nauc_ndcg_at_10_std + value: 8.934209252745864 + - type: nauc_ndcg_at_1_diff1 + value: 24.868548821073084 + - type: nauc_ndcg_at_1_max + value: 16.189915011439044 + - type: nauc_ndcg_at_1_std + value: -0.17692171251799987 + - type: nauc_ndcg_at_20_diff1 + value: 16.647406628819976 + - type: nauc_ndcg_at_20_max + value: 25.64488140369063 + - type: nauc_ndcg_at_20_std + value: 10.587157641309098 + - type: nauc_ndcg_at_3_diff1 + value: 19.093302254257377 + - type: nauc_ndcg_at_3_max + value: 21.33725971448413 + - type: nauc_ndcg_at_3_std + value: 2.549021710462978 + - type: nauc_ndcg_at_5_diff1 + value: 19.495189389728836 + - type: nauc_ndcg_at_5_max + value: 24.21965138651894 + - type: nauc_ndcg_at_5_std + value: 5.549408503444251 + - type: nauc_precision_at_1000_diff1 + value: 7.4232833098081565 + - type: nauc_precision_at_1000_max + value: 25.24619675919913 + - type: nauc_precision_at_1000_std + value: 32.79744946411614 + - type: nauc_precision_at_100_diff1 + value: 10.550449529674747 + - type: nauc_precision_at_100_max + value: 25.652112631579726 + - type: nauc_precision_at_100_std + value: 26.65722909800614 + - type: nauc_precision_at_10_diff1 + value: 11.195653785882708 + - type: nauc_precision_at_10_max + value: 26.469986306854977 + - type: nauc_precision_at_10_std + value: 14.05089697514966 + - type: nauc_precision_at_1_diff1 + value: 24.868548821073084 + - type: nauc_precision_at_1_max + value: 16.189915011439044 + - type: nauc_precision_at_1_std + value: -0.17692171251799987 + - type: nauc_precision_at_20_diff1 + value: 11.16738184991032 + - type: nauc_precision_at_20_max + value: 26.53741675130711 + - type: nauc_precision_at_20_std + value: 16.250110771034542 + - type: nauc_precision_at_3_diff1 + value: 16.917872510926284 + - type: nauc_precision_at_3_max + value: 23.22094310791854 + - type: nauc_precision_at_3_std + value: 3.9255078517383906 + - type: nauc_precision_at_5_diff1 + value: 16.898056883587824 + - type: nauc_precision_at_5_max + value: 27.39457295203392 + - type: nauc_precision_at_5_std + value: 8.924759582566171 + - type: nauc_recall_at_1000_diff1 + value: 7.516072705946253 + - type: nauc_recall_at_1000_max + value: 25.001682297424594 + - type: nauc_recall_at_1000_std + value: 33.86296283879721 + - type: nauc_recall_at_100_diff1 + value: 10.435705067998168 + - type: nauc_recall_at_100_max + value: 25.31622603650995 + - type: nauc_recall_at_100_std + value: 26.758897185352097 + - type: nauc_recall_at_10_diff1 + value: 11.110953419292343 + - type: nauc_recall_at_10_max + value: 25.970593144433085 + - type: nauc_recall_at_10_std + value: 13.92252981022314 + - type: nauc_recall_at_1_diff1 + value: 24.9292922418417 + - type: nauc_recall_at_1_max + value: 15.49253311874767 + - type: nauc_recall_at_1_std + value: -0.4754734108717385 + - type: nauc_recall_at_20_diff1 + value: 11.050515317424548 + - type: nauc_recall_at_20_max + value: 26.068866115743134 + - type: nauc_recall_at_20_std + value: 16.13787216291987 + - type: nauc_recall_at_3_diff1 + value: 17.013383740580203 + - type: nauc_recall_at_3_max + value: 22.49105285578937 + - type: nauc_recall_at_3_std + value: 3.5741090487413687 + - type: nauc_recall_at_5_diff1 + value: 16.973540662242602 + - type: nauc_recall_at_5_max + value: 26.78087164318061 + - type: nauc_recall_at_5_std + value: 8.68040862354009 + - type: ndcg_at_1 + value: 23.799999999999997 + - type: ndcg_at_10 + value: 20.86 + - type: ndcg_at_100 + value: 29.145 + - type: ndcg_at_1000 + value: 34.518 + - type: ndcg_at_20 + value: 23.892 + - type: ndcg_at_3 + value: 19.541 + - type: ndcg_at_5 + value: 17.166999999999998 + - type: precision_at_1 + value: 23.799999999999997 + - type: precision_at_10 + value: 10.9 + - type: precision_at_100 + value: 2.281 + - type: precision_at_1000 + value: 0.357 + - type: precision_at_20 + value: 7.21 + - type: precision_at_3 + value: 18.3 + - type: precision_at_5 + value: 15.120000000000001 + - type: recall_at_1 + value: 4.843 + - type: recall_at_10 + value: 22.12 + - type: recall_at_100 + value: 46.257 + - type: recall_at_1000 + value: 72.382 + - type: recall_at_20 + value: 29.253 + - type: recall_at_3 + value: 11.158 + - type: recall_at_5 + value: 15.347 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SICK-R (default) + revision: 20a6d6f312dd54037fe07a32d58e5e168867909d + split: test + type: mteb/sickr-sts + metrics: + - type: cosine_pearson + value: 85.65467419133347 + - type: cosine_spearman + value: 81.88046945336663 + - type: euclidean_pearson + value: 82.82887106181879 + - type: euclidean_spearman + value: 81.88047605481775 + - type: main_score + value: 81.88046945336663 + - type: manhattan_pearson + value: 82.7839019603756 + - type: manhattan_spearman + value: 81.83505450284663 + - type: pearson + value: 85.65467419133347 + - type: spearman + value: 81.88046945336663 + task: + type: STS + - dataset: + config: default + name: MTEB STS12 (default) + revision: a0d554a64d88156834ff5ae9920b964011b16384 + split: test + type: mteb/sts12-sts + metrics: + - type: cosine_pearson + value: 85.8979872663498 + - type: cosine_spearman + value: 78.63991285161867 + - type: euclidean_pearson + value: 81.20243176386163 + - type: euclidean_spearman + value: 78.64021127260493 + - type: main_score + value: 78.63991285161867 + - type: manhattan_pearson + value: 81.58673652635328 + - type: manhattan_spearman + value: 79.03930665482164 + - type: pearson + value: 85.8979872663498 + - type: spearman + value: 78.63991285161867 + task: + type: STS + - dataset: + config: default + name: MTEB STS13 (default) + revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca + split: test + type: mteb/sts13-sts + metrics: + - type: cosine_pearson + value: 87.10598414063074 + - type: cosine_spearman + value: 87.12110799581852 + - type: euclidean_pearson + value: 86.52284239759508 + - type: euclidean_spearman + value: 87.12110799581852 + - type: main_score + value: 87.12110799581852 + - type: manhattan_pearson + value: 86.61105352996736 + - type: manhattan_spearman + value: 87.34100209521596 + - type: pearson + value: 87.10598414063074 + - type: spearman + value: 87.12110799581852 + task: + type: STS + - dataset: + config: default + name: MTEB STS14 (default) + revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 + split: test + type: mteb/sts14-sts + metrics: + - type: cosine_pearson + value: 85.66540041627184 + - type: cosine_spearman + value: 83.55263671417923 + - type: euclidean_pearson + value: 84.2332532036626 + - type: euclidean_spearman + value: 83.55264421653584 + - type: main_score + value: 83.55263671417923 + - type: manhattan_pearson + value: 84.14418954784165 + - type: manhattan_spearman + value: 83.58193360267302 + - type: pearson + value: 85.66540041627184 + - type: spearman + value: 83.55263671417923 + task: + type: STS + - dataset: + config: default + name: MTEB STS15 (default) + revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 + split: test + type: mteb/sts15-sts + metrics: + - type: cosine_pearson + value: 89.83404956912175 + - type: cosine_spearman + value: 90.09569633194636 + - type: euclidean_pearson + value: 89.31121256629982 + - type: euclidean_spearman + value: 90.09569632193572 + - type: main_score + value: 90.09569633194636 + - type: manhattan_pearson + value: 89.30064909066367 + - type: manhattan_spearman + value: 90.20232732019451 + - type: pearson + value: 89.83404956912175 + - type: spearman + value: 90.09569633194636 + task: + type: STS + - dataset: + config: default + name: MTEB STS16 (default) + revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 + split: test + type: mteb/sts16-sts + metrics: + - type: cosine_pearson + value: 86.27894370732598 + - type: cosine_spearman + value: 87.22000226558832 + - type: euclidean_pearson + value: 85.92822715155758 + - type: euclidean_spearman + value: 87.22000226558832 + - type: main_score + value: 87.22000226558832 + - type: manhattan_pearson + value: 85.9498561399522 + - type: manhattan_spearman + value: 87.28837300894288 + - type: pearson + value: 86.27894370732598 + - type: spearman + value: 87.22000226558832 + task: + type: STS + - dataset: + config: it-en + name: MTEB STS17 (it-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 88.7349764804212 + - type: cosine_spearman + value: 89.28951442654947 + - type: euclidean_pearson + value: 88.88291732710609 + - type: euclidean_spearman + value: 89.28951442654947 + - type: main_score + value: 89.28951442654947 + - type: manhattan_pearson + value: 88.98193254738537 + - type: manhattan_spearman + value: 89.24257960766334 + - type: pearson + value: 88.7349764804212 + - type: spearman + value: 89.28951442654947 + task: + type: STS + - dataset: + config: en-ar + name: MTEB STS17 (en-ar) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 77.0392141246108 + - type: cosine_spearman + value: 77.59284352641623 + - type: euclidean_pearson + value: 77.0732057512901 + - type: euclidean_spearman + value: 77.59284352641623 + - type: main_score + value: 77.59284352641623 + - type: manhattan_pearson + value: 77.20818950201269 + - type: manhattan_spearman + value: 77.47925364624454 + - type: pearson + value: 77.0392141246108 + - type: spearman + value: 77.59284352641623 + task: + type: STS + - dataset: + config: en-tr + name: MTEB STS17 (en-tr) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 73.88921691413196 + - type: cosine_spearman + value: 72.16578879610319 + - type: euclidean_pearson + value: 73.97983260470556 + - type: euclidean_spearman + value: 72.16578879610319 + - type: main_score + value: 72.16578879610319 + - type: manhattan_pearson + value: 73.99943159820614 + - type: manhattan_spearman + value: 72.21219442454583 + - type: pearson + value: 73.88921691413196 + - type: spearman + value: 72.16578879610319 + task: + type: STS + - dataset: + config: es-en + name: MTEB STS17 (es-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 86.97186290205772 + - type: cosine_spearman + value: 87.93811587095317 + - type: euclidean_pearson + value: 87.23624630702996 + - type: euclidean_spearman + value: 87.93811587095317 + - type: main_score + value: 87.93811587095317 + - type: manhattan_pearson + value: 87.72832812198037 + - type: manhattan_spearman + value: 88.49315823056962 + - type: pearson + value: 86.97186290205772 + - type: spearman + value: 87.93811587095317 + task: + type: STS + - dataset: + config: en-en + name: MTEB STS17 (en-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 91.60185356782324 + - type: cosine_spearman + value: 91.43471625912765 + - type: euclidean_pearson + value: 91.52529087606635 + - type: euclidean_spearman + value: 91.43471625912765 + - type: main_score + value: 91.43471625912765 + - type: manhattan_pearson + value: 91.34917173506308 + - type: manhattan_spearman + value: 91.2112665439884 + - type: pearson + value: 91.60185356782324 + - type: spearman + value: 91.43471625912765 + task: + type: STS + - dataset: + config: en-de + name: MTEB STS17 (en-de) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 87.07489674877769 + - type: cosine_spearman + value: 87.22603530072956 + - type: euclidean_pearson + value: 87.46339210182721 + - type: euclidean_spearman + value: 87.22603530072956 + - type: main_score + value: 87.22603530072956 + - type: manhattan_pearson + value: 87.60737696451993 + - type: manhattan_spearman + value: 87.16064944739192 + - type: pearson + value: 87.07489674877769 + - type: spearman + value: 87.22603530072956 + task: + type: STS + - dataset: + config: nl-en + name: MTEB STS17 (nl-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 87.90335800122372 + - type: cosine_spearman + value: 88.26559733345387 + - type: euclidean_pearson + value: 88.03567118672723 + - type: euclidean_spearman + value: 88.26559733345387 + - type: main_score + value: 88.26559733345387 + - type: manhattan_pearson + value: 88.04841463079502 + - type: manhattan_spearman + value: 88.21428039388552 + - type: pearson + value: 87.90335800122372 + - type: spearman + value: 88.26559733345387 + task: + type: STS + - dataset: + config: fr-en + name: MTEB STS17 (fr-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 87.78539484170877 + - type: cosine_spearman + value: 87.80101566182594 + - type: euclidean_pearson + value: 87.98116016872204 + - type: euclidean_spearman + value: 87.80101566182594 + - type: main_score + value: 87.80101566182594 + - type: manhattan_pearson + value: 87.96905768863878 + - type: manhattan_spearman + value: 87.89207920477536 + - type: pearson + value: 87.78539484170877 + - type: spearman + value: 87.80101566182594 + task: + type: STS + - dataset: + config: de-en + name: MTEB STS22 (de-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 65.74027985098283 + - type: cosine_spearman + value: 60.72082677272566 + - type: euclidean_pearson + value: 67.81412578640078 + - type: euclidean_spearman + value: 60.72082677272566 + - type: main_score + value: 60.72082677272566 + - type: manhattan_pearson + value: 68.4634050117408 + - type: manhattan_spearman + value: 59.13639592328217 + - type: pearson + value: 65.74027985098283 + - type: spearman + value: 60.72082677272566 + task: + type: STS + - dataset: + config: pl-en + name: MTEB STS22 (pl-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 77.50079791702605 + - type: cosine_spearman + value: 75.31642718290283 + - type: euclidean_pearson + value: 77.23908769580555 + - type: euclidean_spearman + value: 75.31642718290283 + - type: main_score + value: 75.31642718290283 + - type: manhattan_pearson + value: 76.69583932932362 + - type: manhattan_spearman + value: 74.28498074617474 + - type: pearson + value: 77.50079791702605 + - type: spearman + value: 75.31642718290283 + task: + type: STS + - dataset: + config: es-en + name: MTEB STS22 (es-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 77.36402411244642 + - type: cosine_spearman + value: 78.26932923484236 + - type: euclidean_pearson + value: 77.71590681654918 + - type: euclidean_spearman + value: 78.26932923484236 + - type: main_score + value: 78.26932923484236 + - type: manhattan_pearson + value: 78.49147427458551 + - type: manhattan_spearman + value: 78.95794678007539 + - type: pearson + value: 77.36402411244642 + - type: spearman + value: 78.26932923484236 + task: + type: STS + - dataset: + config: en + name: MTEB STS22 (en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 68.735098373629 + - type: cosine_spearman + value: 67.76156085991387 + - type: euclidean_pearson + value: 68.38053954511516 + - type: euclidean_spearman + value: 67.76156085991387 + - type: main_score + value: 67.76156085991387 + - type: manhattan_pearson + value: 68.4533080173714 + - type: manhattan_spearman + value: 67.76676959397871 + - type: pearson + value: 68.735098373629 + - type: spearman + value: 67.76156085991387 + task: + type: STS + - dataset: + config: zh-en + name: MTEB STS22 (zh-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 73.6098575655897 + - type: cosine_spearman + value: 72.42424022416704 + - type: euclidean_pearson + value: 74.38100861043216 + - type: euclidean_spearman + value: 72.42424022416704 + - type: main_score + value: 72.42424022416704 + - type: manhattan_pearson + value: 75.48257861401288 + - type: manhattan_spearman + value: 72.88125030576865 + - type: pearson + value: 73.6098575655897 + - type: spearman + value: 72.42424022416704 + task: + type: STS + - dataset: + config: default + name: MTEB STSBenchmark (default) + revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 + split: test + type: mteb/stsbenchmark-sts + metrics: + - type: cosine_pearson + value: 87.63236624274985 + - type: cosine_spearman + value: 88.27561759951514 + - type: euclidean_pearson + value: 87.61137355553329 + - type: euclidean_spearman + value: 88.27561759951514 + - type: main_score + value: 88.27561759951514 + - type: manhattan_pearson + value: 87.63505381780153 + - type: manhattan_spearman + value: 88.41268943146845 + - type: pearson + value: 87.63236624274985 + - type: spearman + value: 88.27561759951514 + task: + type: STS + - dataset: + config: default + name: MTEB SciDocsRR (default) + revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab + split: test + type: mteb/scidocs-reranking + metrics: + - type: main_score + value: 85.16412972900244 + - type: map + value: 85.16412972900244 + - type: mrr + value: 96.15786628041529 + - type: nAUC_map_diff1 + value: -1.5068306084088756 + - type: nAUC_map_max + value: 48.81296049442589 + - type: nAUC_map_std + value: 65.0187132933644 + - type: nAUC_mrr_diff1 + value: 44.22872564939586 + - type: nAUC_mrr_max + value: 85.19719227096536 + - type: nAUC_mrr_std + value: 79.62669870868876 + task: + type: Reranking + - dataset: + config: default + name: MTEB SciFact (default) + revision: 0228b52cf27578f30900b9e5271d331663a030d7 + split: test + type: mteb/scifact + metrics: + - type: main_score + value: 77.377 + - type: map_at_1 + value: 62.217 + - type: map_at_10 + value: 73.115 + - type: map_at_100 + value: 73.63499999999999 + - type: map_at_1000 + value: 73.644 + - type: map_at_20 + value: 73.528 + - type: map_at_3 + value: 70.62 + - type: map_at_5 + value: 72.16 + - type: mrr_at_1 + value: 65.0 + - type: mrr_at_10 + value: 74.05542328042326 + - type: mrr_at_100 + value: 74.46295785951277 + - type: mrr_at_1000 + value: 74.47168088874803 + - type: mrr_at_20 + value: 74.35632423132421 + - type: mrr_at_3 + value: 72.55555555555556 + - type: mrr_at_5 + value: 73.38888888888887 + - type: nauc_map_at_1000_diff1 + value: 72.31754010838618 + - type: nauc_map_at_1000_max + value: 60.59518156728312 + - type: nauc_map_at_1000_std + value: -3.601504782295705 + - type: nauc_map_at_100_diff1 + value: 72.32057771059107 + - type: nauc_map_at_100_max + value: 60.60481879601873 + - type: nauc_map_at_100_std + value: -3.6030430073837167 + - type: nauc_map_at_10_diff1 + value: 72.15009895006031 + - type: nauc_map_at_10_max + value: 60.49958178006608 + - type: nauc_map_at_10_std + value: -4.305475753173601 + - type: nauc_map_at_1_diff1 + value: 76.32919417574946 + - type: nauc_map_at_1_max + value: 54.77358788281581 + - type: nauc_map_at_1_std + value: -9.773898055794557 + - type: nauc_map_at_20_diff1 + value: 72.15740734516393 + - type: nauc_map_at_20_max + value: 60.61318821265446 + - type: nauc_map_at_20_std + value: -3.6016854193910803 + - type: nauc_map_at_3_diff1 + value: 72.07435404889445 + - type: nauc_map_at_3_max + value: 56.93970890047747 + - type: nauc_map_at_3_std + value: -8.697324220121793 + - type: nauc_map_at_5_diff1 + value: 72.42599960854554 + - type: nauc_map_at_5_max + value: 60.12535137001906 + - type: nauc_map_at_5_std + value: -4.437892354037166 + - type: nauc_mrr_at_1000_diff1 + value: 72.75103842052889 + - type: nauc_mrr_at_1000_max + value: 62.72341811793062 + - type: nauc_mrr_at_1000_std + value: -0.7759889099766357 + - type: nauc_mrr_at_100_diff1 + value: 72.75396801842608 + - type: nauc_mrr_at_100_max + value: 62.73241247525427 + - type: nauc_mrr_at_100_std + value: -0.7786866224468205 + - type: nauc_mrr_at_10_diff1 + value: 72.5942754009733 + - type: nauc_mrr_at_10_max + value: 62.895066542256664 + - type: nauc_mrr_at_10_std + value: -0.9018200301159104 + - type: nauc_mrr_at_1_diff1 + value: 77.63311362465076 + - type: nauc_mrr_at_1_max + value: 62.42059294219759 + - type: nauc_mrr_at_1_std + value: -1.3182520953698476 + - type: nauc_mrr_at_20_diff1 + value: 72.58522880943326 + - type: nauc_mrr_at_20_max + value: 62.73063935403417 + - type: nauc_mrr_at_20_std + value: -0.7910003366564456 + - type: nauc_mrr_at_3_diff1 + value: 72.70751722757556 + - type: nauc_mrr_at_3_max + value: 62.38218933726893 + - type: nauc_mrr_at_3_std + value: -1.7126398606397155 + - type: nauc_mrr_at_5_diff1 + value: 72.57550761997256 + - type: nauc_mrr_at_5_max + value: 62.70945847818393 + - type: nauc_mrr_at_5_std + value: -0.30886077098332143 + - type: nauc_ndcg_at_1000_diff1 + value: 71.6036105202873 + - type: nauc_ndcg_at_1000_max + value: 61.99911514670603 + - type: nauc_ndcg_at_1000_std + value: -2.050470755577302 + - type: nauc_ndcg_at_100_diff1 + value: 71.70345074974581 + - type: nauc_ndcg_at_100_max + value: 62.374525611545714 + - type: nauc_ndcg_at_100_std + value: -1.922345118135967 + - type: nauc_ndcg_at_10_diff1 + value: 70.40027928749286 + - type: nauc_ndcg_at_10_max + value: 62.36595526966657 + - type: nauc_ndcg_at_10_std + value: -3.862278807246422 + - type: nauc_ndcg_at_1_diff1 + value: 77.63311362465076 + - type: nauc_ndcg_at_1_max + value: 62.42059294219759 + - type: nauc_ndcg_at_1_std + value: -1.3182520953698476 + - type: nauc_ndcg_at_20_diff1 + value: 70.21719291674641 + - type: nauc_ndcg_at_20_max + value: 62.356711760569404 + - type: nauc_ndcg_at_20_std + value: -2.240360396463778 + - type: nauc_ndcg_at_3_diff1 + value: 70.72483260039468 + - type: nauc_ndcg_at_3_max + value: 59.465348910073445 + - type: nauc_ndcg_at_3_std + value: -6.379991854598364 + - type: nauc_ndcg_at_5_diff1 + value: 70.91296936044013 + - type: nauc_ndcg_at_5_max + value: 61.5986283773017 + - type: nauc_ndcg_at_5_std + value: -3.064893399445654 + - type: nauc_precision_at_1000_diff1 + value: -25.399544557043956 + - type: nauc_precision_at_1000_max + value: 17.838641101318792 + - type: nauc_precision_at_1000_std + value: 54.531382221213185 + - type: nauc_precision_at_100_diff1 + value: -15.78139909072201 + - type: nauc_precision_at_100_max + value: 24.183801380755472 + - type: nauc_precision_at_100_std + value: 50.39320972640593 + - type: nauc_precision_at_10_diff1 + value: 4.1199958514831 + - type: nauc_precision_at_10_max + value: 37.922630159717926 + - type: nauc_precision_at_10_std + value: 32.94959551960178 + - type: nauc_precision_at_1_diff1 + value: 77.63311362465076 + - type: nauc_precision_at_1_max + value: 62.42059294219759 + - type: nauc_precision_at_1_std + value: -1.3182520953698476 + - type: nauc_precision_at_20_diff1 + value: -8.926047159112303 + - type: nauc_precision_at_20_max + value: 29.369903951067172 + - type: nauc_precision_at_20_std + value: 41.793379234725904 + - type: nauc_precision_at_3_diff1 + value: 36.51209832895358 + - type: nauc_precision_at_3_max + value: 51.07398992745159 + - type: nauc_precision_at_3_std + value: 13.831661495933623 + - type: nauc_precision_at_5_diff1 + value: 19.526084047733807 + - type: nauc_precision_at_5_max + value: 46.67537950098273 + - type: nauc_precision_at_5_std + value: 31.06747779005178 + - type: nauc_recall_at_1000_diff1 + value: .nan + - type: nauc_recall_at_1000_max + value: .nan + - type: nauc_recall_at_1000_std + value: .nan + - type: nauc_recall_at_100_diff1 + value: 77.45764972655711 + - type: nauc_recall_at_100_max + value: 85.69427771108462 + - type: nauc_recall_at_100_std + value: 10.277444311057575 + - type: nauc_recall_at_10_diff1 + value: 59.14685653975806 + - type: nauc_recall_at_10_max + value: 67.75739956082005 + - type: nauc_recall_at_10_std + value: -12.22251646924215 + - type: nauc_recall_at_1_diff1 + value: 76.32919417574946 + - type: nauc_recall_at_1_max + value: 54.77358788281581 + - type: nauc_recall_at_1_std + value: -9.773898055794557 + - type: nauc_recall_at_20_diff1 + value: 49.90908644159423 + - type: nauc_recall_at_20_max + value: 70.55383556931541 + - type: nauc_recall_at_20_std + value: -3.7004275394368182 + - type: nauc_recall_at_3_diff1 + value: 64.34183819693267 + - type: nauc_recall_at_3_max + value: 55.782789721196444 + - type: nauc_recall_at_3_std + value: -13.886583892174077 + - type: nauc_recall_at_5_diff1 + value: 63.467364565196135 + - type: nauc_recall_at_5_max + value: 62.51562390716315 + - type: nauc_recall_at_5_std + value: -4.715416491952255 + - type: ndcg_at_1 + value: 65.0 + - type: ndcg_at_10 + value: 77.377 + - type: ndcg_at_100 + value: 79.36 + - type: ndcg_at_1000 + value: 79.644 + - type: ndcg_at_20 + value: 78.61200000000001 + - type: ndcg_at_3 + value: 73.624 + - type: ndcg_at_5 + value: 75.458 + - type: precision_at_1 + value: 65.0 + - type: precision_at_10 + value: 10.100000000000001 + - type: precision_at_100 + value: 1.107 + - type: precision_at_1000 + value: 0.11299999999999999 + - type: precision_at_20 + value: 5.333 + - type: precision_at_3 + value: 28.778 + - type: precision_at_5 + value: 18.8 + - type: recall_at_1 + value: 62.217 + - type: recall_at_10 + value: 89.156 + - type: recall_at_100 + value: 97.667 + - type: recall_at_1000 + value: 100.0 + - type: recall_at_20 + value: 93.667 + - type: recall_at_3 + value: 79.183 + - type: recall_at_5 + value: 83.672 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SprintDuplicateQuestions (default) + revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 + split: test + type: mteb/sprintduplicatequestions-pairclassification + metrics: + - type: ap + value: 95.25139396923107 + - type: main_score + value: 95.43878917155146 + task: + type: PairClassification + - dataset: + config: default + name: MTEB StackExchangeClustering (default) + revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 + split: test + type: mteb/stackexchange-clustering + metrics: + - type: main_score + value: 75.91311883393888 + - type: v_measure + value: 75.91311883393888 + - type: v_measure_std + value: 3.286198100593212 + task: + type: Clustering + - dataset: + config: default + name: MTEB StackExchangeClusteringP2P (default) + revision: 815ca46b2622cec33ccafc3735d572c266efdb44 + split: test + type: mteb/stackexchange-clustering-p2p + metrics: + - type: main_score + value: 47.171049215275694 + - type: v_measure + value: 47.171049215275694 + - type: v_measure_std + value: 1.6586563477857534 + task: + type: Clustering + - dataset: + config: default + name: MTEB StackOverflowDupQuestions (default) + revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 + split: test + type: mteb/stackoverflowdupquestions-reranking + metrics: + - type: main_score + value: 54.15041943470163 + - type: map + value: 54.15041943470163 + - type: mrr + value: 55.03112798149563 + - type: nAUC_map_diff1 + value: 39.50144777017669 + - type: nAUC_map_max + value: 14.024793174481395 + - type: nAUC_map_std + value: 6.533766502190137 + - type: nAUC_mrr_diff1 + value: 39.72560651870919 + - type: nAUC_mrr_max + value: 14.807887392821616 + - type: nAUC_mrr_std + value: 7.270272018791473 + task: + type: Reranking + - dataset: + config: default + name: MTEB SummEval (default) + revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c + split: test + type: mteb/summeval + metrics: + - type: cosine_pearson + value: 31.57515534177576 + - type: cosine_spearman + value: 31.415247541636194 + - type: dot_pearson + value: 31.575170220667488 + - type: dot_spearman + value: 31.415247541636194 + - type: main_score + value: 31.415247541636194 + - type: pearson + value: 31.57515534177576 + - type: spearman + value: 31.415247541636194 + task: + type: Summarization + - dataset: + config: default + name: MTEB TRECCOVID (default) + revision: bb9466bac8153a0349341eb1b22e06409e78ef4e + split: test + type: mteb/trec-covid + metrics: + - type: main_score + value: 83.67999999999999 + - type: map_at_1 + value: 0.243 + - type: map_at_10 + value: 2.167 + - type: map_at_100 + value: 13.750000000000002 + - type: map_at_1000 + value: 33.537 + - type: map_at_20 + value: 4.047 + - type: map_at_3 + value: 0.694 + - type: map_at_5 + value: 1.141 + - type: mrr_at_1 + value: 94.0 + - type: mrr_at_10 + value: 97.0 + - type: mrr_at_100 + value: 97.0 + - type: mrr_at_1000 + value: 97.0 + - type: mrr_at_20 + value: 97.0 + - type: mrr_at_3 + value: 97.0 + - type: mrr_at_5 + value: 97.0 + - type: nauc_map_at_1000_diff1 + value: 4.890354942949616 + - type: nauc_map_at_1000_max + value: 29.279958833328408 + - type: nauc_map_at_1000_std + value: 77.2405348865942 + - type: nauc_map_at_100_diff1 + value: 31.835069149380868 + - type: nauc_map_at_100_max + value: 14.523120708509271 + - type: nauc_map_at_100_std + value: 39.682149025882886 + - type: nauc_map_at_10_diff1 + value: 43.45574726953753 + - type: nauc_map_at_10_max + value: -2.9143965183484246 + - type: nauc_map_at_10_std + value: 2.8052301238653756 + - type: nauc_map_at_1_diff1 + value: 26.134637426782753 + - type: nauc_map_at_1_max + value: -3.108959317897608 + - type: nauc_map_at_1_std + value: -5.781123480253076 + - type: nauc_map_at_20_diff1 + value: 45.735224340099236 + - type: nauc_map_at_20_max + value: -1.099022132339708 + - type: nauc_map_at_20_std + value: 7.6378546013151905 + - type: nauc_map_at_3_diff1 + value: 35.70649469812688 + - type: nauc_map_at_3_max + value: -9.710213033638656 + - type: nauc_map_at_3_std + value: -3.6668161574691056 + - type: nauc_map_at_5_diff1 + value: 37.6110093992781 + - type: nauc_map_at_5_max + value: -8.6295080300384 + - type: nauc_map_at_5_std + value: -3.2709712613287145 + - type: nauc_mrr_at_1000_diff1 + value: -22.362278244631675 + - type: nauc_mrr_at_1000_max + value: 63.74105197634592 + - type: nauc_mrr_at_1000_std + value: 69.88795518207282 + - type: nauc_mrr_at_100_diff1 + value: -22.362278244631675 + - type: nauc_mrr_at_100_max + value: 63.74105197634592 + - type: nauc_mrr_at_100_std + value: 69.88795518207282 + - type: nauc_mrr_at_10_diff1 + value: -22.362278244631675 + - type: nauc_mrr_at_10_max + value: 63.74105197634592 + - type: nauc_mrr_at_10_std + value: 69.88795518207282 + - type: nauc_mrr_at_1_diff1 + value: -22.36227824463097 + - type: nauc_mrr_at_1_max + value: 63.741051976346206 + - type: nauc_mrr_at_1_std + value: 69.88795518207289 + - type: nauc_mrr_at_20_diff1 + value: -22.362278244631675 + - type: nauc_mrr_at_20_max + value: 63.74105197634592 + - type: nauc_mrr_at_20_std + value: 69.88795518207282 + - type: nauc_mrr_at_3_diff1 + value: -22.362278244631675 + - type: nauc_mrr_at_3_max + value: 63.74105197634592 + - type: nauc_mrr_at_3_std + value: 69.88795518207282 + - type: nauc_mrr_at_5_diff1 + value: -22.362278244631675 + - type: nauc_mrr_at_5_max + value: 63.74105197634592 + - type: nauc_mrr_at_5_std + value: 69.88795518207282 + - type: nauc_ndcg_at_1000_diff1 + value: 11.950362559089744 + - type: nauc_ndcg_at_1000_max + value: 27.0707842379056 + - type: nauc_ndcg_at_1000_std + value: 72.43903405163071 + - type: nauc_ndcg_at_100_diff1 + value: -3.597031398660954 + - type: nauc_ndcg_at_100_max + value: 24.415981061123944 + - type: nauc_ndcg_at_100_std + value: 74.01146007854192 + - type: nauc_ndcg_at_10_diff1 + value: 17.368676394860337 + - type: nauc_ndcg_at_10_max + value: 27.014276985741652 + - type: nauc_ndcg_at_10_std + value: 50.032884783457476 + - type: nauc_ndcg_at_1_diff1 + value: 5.824544582933801 + - type: nauc_ndcg_at_1_max + value: 39.22818791946299 + - type: nauc_ndcg_at_1_std + value: 29.32406519654831 + - type: nauc_ndcg_at_20_diff1 + value: 17.816409720909615 + - type: nauc_ndcg_at_20_max + value: 25.056392180259827 + - type: nauc_ndcg_at_20_std + value: 58.05680238138826 + - type: nauc_ndcg_at_3_diff1 + value: 15.010486876001556 + - type: nauc_ndcg_at_3_max + value: 4.023535837214374 + - type: nauc_ndcg_at_3_std + value: 22.55308565809234 + - type: nauc_ndcg_at_5_diff1 + value: 12.73162605923733 + - type: nauc_ndcg_at_5_max + value: 15.425379568695105 + - type: nauc_ndcg_at_5_std + value: 34.4442400670659 + - type: nauc_precision_at_1000_diff1 + value: -29.218427320110436 + - type: nauc_precision_at_1000_max + value: 29.90719259849769 + - type: nauc_precision_at_1000_std + value: 48.95093300052051 + - type: nauc_precision_at_100_diff1 + value: -6.881054858812464 + - type: nauc_precision_at_100_max + value: 30.388273677316956 + - type: nauc_precision_at_100_std + value: 76.1031398803066 + - type: nauc_precision_at_10_diff1 + value: 24.298416597687574 + - type: nauc_precision_at_10_max + value: 44.38332754799598 + - type: nauc_precision_at_10_std + value: 61.64143369558439 + - type: nauc_precision_at_1_diff1 + value: -22.36227824463097 + - type: nauc_precision_at_1_max + value: 63.741051976346206 + - type: nauc_precision_at_1_std + value: 69.88795518207289 + - type: nauc_precision_at_20_diff1 + value: 21.823848783430545 + - type: nauc_precision_at_20_max + value: 32.815202091292875 + - type: nauc_precision_at_20_std + value: 61.4003619545546 + - type: nauc_precision_at_3_diff1 + value: 7.264709295578332 + - type: nauc_precision_at_3_max + value: 18.088275115082432 + - type: nauc_precision_at_3_std + value: 46.315423001044266 + - type: nauc_precision_at_5_diff1 + value: 19.4281378539196 + - type: nauc_precision_at_5_max + value: 30.042729922926426 + - type: nauc_precision_at_5_std + value: 48.803961503134936 + - type: nauc_recall_at_1000_diff1 + value: 14.078781719704242 + - type: nauc_recall_at_1000_max + value: 24.205288710944746 + - type: nauc_recall_at_1000_std + value: 60.19521883992679 + - type: nauc_recall_at_100_diff1 + value: 34.68620796161708 + - type: nauc_recall_at_100_max + value: 5.862669275470962 + - type: nauc_recall_at_100_std + value: 23.779387105339538 + - type: nauc_recall_at_10_diff1 + value: 41.60859491145645 + - type: nauc_recall_at_10_max + value: -6.060553984265031 + - type: nauc_recall_at_10_std + value: -3.0401474174665597 + - type: nauc_recall_at_1_diff1 + value: 26.134637426782753 + - type: nauc_recall_at_1_max + value: -3.108959317897608 + - type: nauc_recall_at_1_std + value: -5.781123480253076 + - type: nauc_recall_at_20_diff1 + value: 43.884440668985256 + - type: nauc_recall_at_20_max + value: -5.215456096089841 + - type: nauc_recall_at_20_std + value: 0.6346955652816175 + - type: nauc_recall_at_3_diff1 + value: 36.682959590903515 + - type: nauc_recall_at_3_max + value: -14.003318698999372 + - type: nauc_recall_at_3_std + value: -8.732791648435722 + - type: nauc_recall_at_5_diff1 + value: 37.55874033777468 + - type: nauc_recall_at_5_max + value: -11.475194910000303 + - type: nauc_recall_at_5_std + value: -8.24171387960509 + - type: ndcg_at_1 + value: 90.0 + - type: ndcg_at_10 + value: 83.67999999999999 + - type: ndcg_at_100 + value: 66.268 + - type: ndcg_at_1000 + value: 59.95700000000001 + - type: ndcg_at_20 + value: 80.41199999999999 + - type: ndcg_at_3 + value: 86.989 + - type: ndcg_at_5 + value: 85.60600000000001 + - type: precision_at_1 + value: 94.0 + - type: precision_at_10 + value: 87.0 + - type: precision_at_100 + value: 68.10000000000001 + - type: precision_at_1000 + value: 26.404 + - type: precision_at_20 + value: 83.7 + - type: precision_at_3 + value: 91.333 + - type: precision_at_5 + value: 89.60000000000001 + - type: recall_at_1 + value: 0.243 + - type: recall_at_10 + value: 2.307 + - type: recall_at_100 + value: 16.713 + - type: recall_at_1000 + value: 56.433 + - type: recall_at_20 + value: 4.3950000000000005 + - type: recall_at_3 + value: 0.721 + - type: recall_at_5 + value: 1.194 + task: + type: Retrieval + - dataset: + config: default + name: MTEB Touche2020 (default) + revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f + split: test + type: mteb/touche2020 + metrics: + - type: main_score + value: 27.095999999999997 + - type: map_at_1 + value: 2.708 + - type: map_at_10 + value: 10.926 + - type: map_at_100 + value: 17.023 + - type: map_at_1000 + value: 18.802 + - type: map_at_20 + value: 14.075 + - type: map_at_3 + value: 6.213 + - type: map_at_5 + value: 8.399 + - type: mrr_at_1 + value: 38.775510204081634 + - type: mrr_at_10 + value: 55.96938775510204 + - type: mrr_at_100 + value: 56.566806209663355 + - type: mrr_at_1000 + value: 56.586429443572314 + - type: mrr_at_20 + value: 56.566806209663355 + - type: mrr_at_3 + value: 53.06122448979592 + - type: mrr_at_5 + value: 54.48979591836734 + - type: nauc_map_at_1000_diff1 + value: -3.7447722422200505 + - type: nauc_map_at_1000_max + value: -21.154942580599432 + - type: nauc_map_at_1000_std + value: -3.3769353126366854 + - type: nauc_map_at_100_diff1 + value: -4.211734469956019 + - type: nauc_map_at_100_max + value: -20.97390043676955 + - type: nauc_map_at_100_std + value: -7.108253122379712 + - type: nauc_map_at_10_diff1 + value: -2.503617891657346 + - type: nauc_map_at_10_max + value: -19.76603379959943 + - type: nauc_map_at_10_std + value: -24.813694071646186 + - type: nauc_map_at_1_diff1 + value: -0.6946291628228135 + - type: nauc_map_at_1_max + value: -27.928780525228326 + - type: nauc_map_at_1_std + value: -26.644256007057386 + - type: nauc_map_at_20_diff1 + value: -8.140689983350077 + - type: nauc_map_at_20_max + value: -21.331762857202346 + - type: nauc_map_at_20_std + value: -18.46503512945984 + - type: nauc_map_at_3_diff1 + value: 1.4806459479634606 + - type: nauc_map_at_3_max + value: -20.57096686541149 + - type: nauc_map_at_3_std + value: -27.53855079505183 + - type: nauc_map_at_5_diff1 + value: -2.4911557022868833 + - type: nauc_map_at_5_max + value: -18.468614237544944 + - type: nauc_map_at_5_std + value: -27.422000270609885 + - type: nauc_mrr_at_1000_diff1 + value: 0.5901324153382446 + - type: nauc_mrr_at_1000_max + value: -29.43201840557888 + - type: nauc_mrr_at_1000_std + value: -22.113570283308878 + - type: nauc_mrr_at_100_diff1 + value: 0.6140852308779037 + - type: nauc_mrr_at_100_max + value: -29.423158073762274 + - type: nauc_mrr_at_100_std + value: -22.03830742373018 + - type: nauc_mrr_at_10_diff1 + value: 1.4017303142295798 + - type: nauc_mrr_at_10_max + value: -29.96128226635445 + - type: nauc_mrr_at_10_std + value: -21.182800337655188 + - type: nauc_mrr_at_1_diff1 + value: 2.9967188734445642 + - type: nauc_mrr_at_1_max + value: -28.076201809234135 + - type: nauc_mrr_at_1_std + value: -23.829475793931397 + - type: nauc_mrr_at_20_diff1 + value: 0.6140852308779037 + - type: nauc_mrr_at_20_max + value: -29.423158073762274 + - type: nauc_mrr_at_20_std + value: -22.03830742373018 + - type: nauc_mrr_at_3_diff1 + value: -1.7324100961545983 + - type: nauc_mrr_at_3_max + value: -31.25504536750873 + - type: nauc_mrr_at_3_std + value: -27.693245095141595 + - type: nauc_mrr_at_5_diff1 + value: 0.9366378266246876 + - type: nauc_mrr_at_5_max + value: -28.61911855691654 + - type: nauc_mrr_at_5_std + value: -23.51734198003236 + - type: nauc_ndcg_at_1000_diff1 + value: 5.589806586986813 + - type: nauc_ndcg_at_1000_max + value: -25.54091728191453 + - type: nauc_ndcg_at_1000_std + value: 18.867289766624364 + - type: nauc_ndcg_at_100_diff1 + value: 5.269555604924481 + - type: nauc_ndcg_at_100_max + value: -25.294068947248 + - type: nauc_ndcg_at_100_std + value: 12.57359579076201 + - type: nauc_ndcg_at_10_diff1 + value: -1.8036041625138828 + - type: nauc_ndcg_at_10_max + value: -23.89433650527811 + - type: nauc_ndcg_at_10_std + value: -18.669805340174104 + - type: nauc_ndcg_at_1_diff1 + value: 1.7320960153524356 + - type: nauc_ndcg_at_1_max + value: -30.98970297820504 + - type: nauc_ndcg_at_1_std + value: -22.039818727732 + - type: nauc_ndcg_at_20_diff1 + value: -7.71266194406333 + - type: nauc_ndcg_at_20_max + value: -28.764052281890564 + - type: nauc_ndcg_at_20_std + value: -14.058766573885803 + - type: nauc_ndcg_at_3_diff1 + value: 3.4222049394447023 + - type: nauc_ndcg_at_3_max + value: -23.010397388596147 + - type: nauc_ndcg_at_3_std + value: -23.917570461776442 + - type: nauc_ndcg_at_5_diff1 + value: 0.4359085390014115 + - type: nauc_ndcg_at_5_max + value: -18.328017574440583 + - type: nauc_ndcg_at_5_std + value: -22.301590122411703 + - type: nauc_precision_at_1000_diff1 + value: 5.705380133328601 + - type: nauc_precision_at_1000_max + value: 29.738757046781583 + - type: nauc_precision_at_1000_std + value: 37.25317043193516 + - type: nauc_precision_at_100_diff1 + value: 18.099479915822755 + - type: nauc_precision_at_100_max + value: 1.039647603335084 + - type: nauc_precision_at_100_std + value: 68.43506311503532 + - type: nauc_precision_at_10_diff1 + value: 1.6010906915801002 + - type: nauc_precision_at_10_max + value: -16.21198992516715 + - type: nauc_precision_at_10_std + value: -10.55666484527 + - type: nauc_precision_at_1_diff1 + value: 2.9967188734445642 + - type: nauc_precision_at_1_max + value: -28.076201809234135 + - type: nauc_precision_at_1_std + value: -23.829475793931397 + - type: nauc_precision_at_20_diff1 + value: -9.646266503089361 + - type: nauc_precision_at_20_max + value: -19.25399592456934 + - type: nauc_precision_at_20_std + value: 4.154373672246843 + - type: nauc_precision_at_3_diff1 + value: 6.468923962729313 + - type: nauc_precision_at_3_max + value: -16.75495139962792 + - type: nauc_precision_at_3_std + value: -24.1555216494731 + - type: nauc_precision_at_5_diff1 + value: 1.89724542441865 + - type: nauc_precision_at_5_max + value: -10.916266272968988 + - type: nauc_precision_at_5_std + value: -19.996228467499165 + - type: nauc_recall_at_1000_diff1 + value: -0.3248897031222208 + - type: nauc_recall_at_1000_max + value: -25.08629526651275 + - type: nauc_recall_at_1000_std + value: 72.42326605733102 + - type: nauc_recall_at_100_diff1 + value: 0.20011224230233096 + - type: nauc_recall_at_100_max + value: -25.71382782994985 + - type: nauc_recall_at_100_std + value: 31.40559917674001 + - type: nauc_recall_at_10_diff1 + value: -7.502107897824034 + - type: nauc_recall_at_10_max + value: -26.197156105779833 + - type: nauc_recall_at_10_std + value: -20.067019662396106 + - type: nauc_recall_at_1_diff1 + value: -0.6946291628228135 + - type: nauc_recall_at_1_max + value: -27.928780525228326 + - type: nauc_recall_at_1_std + value: -26.644256007057386 + - type: nauc_recall_at_20_diff1 + value: -16.829462200879107 + - type: nauc_recall_at_20_max + value: -29.55978083865099 + - type: nauc_recall_at_20_std + value: -11.329177422867945 + - type: nauc_recall_at_3_diff1 + value: -4.487251181022699 + - type: nauc_recall_at_3_max + value: -26.28852595660599 + - type: nauc_recall_at_3_std + value: -30.010933869743877 + - type: nauc_recall_at_5_diff1 + value: -7.4729339604681515 + - type: nauc_recall_at_5_max + value: -22.995431038489112 + - type: nauc_recall_at_5_std + value: -27.623494423158906 + - type: ndcg_at_1 + value: 35.714 + - type: ndcg_at_10 + value: 27.095999999999997 + - type: ndcg_at_100 + value: 37.577 + - type: ndcg_at_1000 + value: 50.234 + - type: ndcg_at_20 + value: 28.706 + - type: ndcg_at_3 + value: 34.808 + - type: ndcg_at_5 + value: 31.657999999999998 + - type: precision_at_1 + value: 38.775999999999996 + - type: precision_at_10 + value: 23.061 + - type: precision_at_100 + value: 7.388 + - type: precision_at_1000 + value: 1.5650000000000002 + - type: precision_at_20 + value: 18.776 + - type: precision_at_3 + value: 36.735 + - type: precision_at_5 + value: 31.429000000000002 + - type: recall_at_1 + value: 2.708 + - type: recall_at_10 + value: 16.645 + - type: recall_at_100 + value: 45.953 + - type: recall_at_1000 + value: 84.553 + - type: recall_at_20 + value: 26.259 + - type: recall_at_3 + value: 7.869 + - type: recall_at_5 + value: 11.166 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ToxicConversationsClassification (default) + revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de + split: test + type: mteb/toxic_conversations_50k + metrics: + - type: accuracy + value: 75.87890625 + - type: ap + value: 16.4629793865873 + - type: ap_weighted + value: 16.4629793865873 + - type: f1 + value: 58.32993265544471 + - type: f1_weighted + value: 80.94360012442658 + - type: main_score + value: 75.87890625 + task: + type: Classification + - dataset: + config: default + name: MTEB TweetSentimentExtractionClassification (default) + revision: d604517c81ca91fe16a244d1248fc021f9ecee7a + split: test + type: mteb/tweet_sentiment_extraction + metrics: + - type: accuracy + value: 65.21788341822298 + - type: f1 + value: 65.00914562845475 + - type: f1_weighted + value: 63.672388825903845 + - type: main_score + value: 65.21788341822298 + task: + type: Classification + - dataset: + config: default + name: MTEB TwentyNewsgroupsClustering (default) + revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 + split: test + type: mteb/twentynewsgroups-clustering + metrics: + - type: main_score + value: 57.152337838073485 + - type: v_measure + value: 57.152337838073485 + - type: v_measure_std + value: 0.8799366494028795 + task: + type: Clustering + - dataset: + config: default + name: MTEB TwitterSemEval2015 (default) + revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 + split: test + type: mteb/twittersemeval2015-pairclassification + metrics: + - type: accuracy + value: 88.63920844012637 + - type: accuracy_threshold + value: 84.5294713973999 + - type: ap + value: 81.41443623323144 + - type: f1 + value: 74.84396892115653 + - type: f1_threshold + value: 82.87262320518494 + - type: precision + value: 72.34671263235656 + - type: recall + value: 77.5197889182058 + - type: main_score + value: 81.41444048232692 + task: + type: PairClassification + - dataset: + config: default + name: MTEB TwitterURLCorpus (default) + revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf + split: test + type: mteb/twitterurlcorpus-pairclassification + metrics: + - type: accuracy + value: 89.90569332867622 + - type: accuracy_threshold + value: 81.01733326911926 + - type: ap + value: 87.91824445175028 + - type: f1 + value: 80.35081439054949 + - type: f1_threshold + value: 78.65387201309204 + - type: precision + value: 75.0853013982739 + - type: recall + value: 86.41053279950724 + - type: main_score + value: 87.91825594329686 + task: + type: PairClassification +language: +- en +license: mit +--- +## Zeta-Alpha-E5-Mistral + +We introduce Zeta Alpha's first public embedding model, a retrieval-specialized, 7B parameter embedding model trained on top of [E5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct). +This model marks the first published model from Zeta Alpha's open science embedding models. The full dataset and detailed training recipe will be made available (very) soon. + +We will also make available our evaluation dataset (Called `Mini` and `Nano` versions of `BEIR`), which allowed us to quickly evaluate the model's performance while training. + + +### Lora Weights + +The lora weights are also available, so there is no need to download the full model. +## How to Run + +The model was trained with the same instruction-tuning strategy as the original [E5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct) model. Therefore, queries should be formatted as follows: + +``` +Instruct: \nQuery: +``` + +### Transformers +``` python +import torch +import torch.nn.functional as F +from torch import Tensor +from transformers import AutoTokenizer, AutoModel + +def last_token_pool(last_hidden_states: Tensor, + attention_mask: Tensor) -> Tensor: + left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0]) + if left_padding: + return last_hidden_states[:, -1] + else: + sequence_lengths = attention_mask.sum(dim=1) - 1 + batch_size = last_hidden_states.shape[0] + return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths] +def get_detailed_instruct(task_description: str, query: str) -> str: + return f'Instruct: {task_description}\nQuery: {query}' + +task = "Given a claim about climate change, retrieve documents that support or refute the claim" +queries = [ + get_detailed_instruct(task, "In Alaska, brown bears are changing their feeding habits to eat elderberries that ripen earlier."), + get_detailed_instruct(task, "Local and regional sea levels continue to exhibit typical natural variability—in some places rising and in others falling.") +] + +passages = [ + "The brown bear ( Ursus arctos ) is a large bear with the widest distribution of any living ursid . The species is distributed across much of northern Eurasia and North America . It is one of the two largest terrestrial carnivorans alive today , rivaled in body size only by its close cousin , the polar bear ( Ursus maritimus ) , which is much less variable in size and averages larger due to this . There are several recognized subspecies , many of which are quite well-known within their native ranges , found in the brown bear species . The brown bear 's principal range includes parts of Russia , Central Asia , China , Canada , the United States ( mostly Alaska ) , Scandinavia and the Carpathian region ( especially Romania ) , Anatolia , and Caucasus . The brown bear is recognized as a national and state animal in several European countries . While the brown bear 's range has shrunk and it has faced local extinctions , it remains listed as a least concern species by the International Union for Conservation of Nature ( IUCN ) with a total population of approximately 200,000 . As of 2012 , this and the American black bear are the only bear species not classified as threatened by the IUCN . However , the Californian , North African ( Atlas bear ) , and Mexican subspecies were hunted to extinction in the nineteenth and early twentieth centuries , and many of the southern Asian subspecies are highly endangered . One of the smaller-bodied subspecies , the Himalayan brown bear , is critically endangered , occupying only 2 % of its former range and threatened by uncontrolled poaching for its parts . The Marsican brown bear , one of several currently isolated populations of the main Eurasian brown bear race , in central Italy is believed to have a population of just 30 to 40 bears .", + "ean sea level ( MSL ) ( abbreviated simply sea level ) is an average level of the surface of one or more of Earth 's oceans from which heights such as elevations may be measured . MSL is a type of vertical datuma standardised geodetic reference pointthat is used , for example , as a chart datum in cartography and marine navigation , or , in aviation , as the standard sea level at which atmospheric pressure is measured in order to calibrate altitude and , consequently , aircraft flight levels . A common and relatively straightforward mean sea-level standard is the midpoint between a mean low and mean high tide at a particular location . Sea levels can be affected by many factors and are known to have varied greatly over geological time scales . The careful measurement of variations in MSL can offer insights into ongoing climate change , and sea level rise has been widely quoted as evidence of ongoing global warming . The term above sea level generally refers to above mean sea level ( AMSL ) ." +] + +# load model and tokenizer +tokenizer = AutoTokenizer.from_pretrained("zeta-alpha-ai/Zeta-Alpha-E5-Mistral") +model = AutoModel.from_pretrained("zeta-alpha-ai/Zeta-Alpha-E5-Mistral") + +max_length = 4096 +input_texts = queries + passages +batch_dict = tokenizer(input_texts, max_length=max_length, padding=True, truncation=True, return_tensors="pt") +outputs = model(**batch_dict) +embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask']) + +# get the embeddings +max_length = 4096 +input_texts = queries + passages +batch_dict = tokenizer(input_texts, max_length=max_length, padding=True, truncation=True, return_tensors="pt") +outputs = model(**batch_dict) +embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask']) + +# normalize embeddings +embeddings = F.normalize(embeddings, p=2, dim=1) +scores = (embeddings[:2] @ embeddings[2:].T) * 100 +print(scores.tolist()) +[[66.15530395507812, 43.65541458129883], [47.681705474853516, 63.67986297607422]] + +``` + +### Zeta Alpha +Zeta Alpha is a the premier Neural Discovery Platform for AI and more. We are an Amsterdam-based R&D and product lab with a passion for AI technology, with offices on the Science Park campus of the University of Amsterdam. and in San Francisco. + + +The Zeta Alpha Research team: +- Arthur Câmara +- Dinos Papakostas +- Mathias Parisot +- Fernando Rejon Barrera +- Jakub Zavrel \ No newline at end of file