--- language: - en license: apache-2.0 tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:6300 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss base_model: BAAI/bge-base-en-v1.5 widget: - source_sentence: What was the total amount of current assets reported by The Hershey Company for the year 2023? sentences: - The total AUS for all categories, including alternative investments, equity, fixed income, and liquidity products, summed up to $2,812 billion in 2023. - The Hershey Company reported a total of current assets amounting to $2,912,103 for the year 2023. - Information on legal proceedings is included in Note 15 to the Consolidated Financial Statements. - source_sentence: What is listed under Item 8 in the document? sentences: - Chubb Limited further advanced their goal of greater product, customer, and geographical diversification with incremental purchases that led to a controlling majority interest in Huatai Insurance Group Co. Ltd, owning about 76.5 percent as of July 1, 2023. - Item 8 includes Financial Statements and Supplementary Data. - Further, state attorneys general may bring civil actions seeking either injunction or an unspecified amount in damages in response to violations of the HIPAA privacy and security regulations. - source_sentence: What were the main factors contributing to the change in net sales for fiscal 2022? sentences: - The decrease in consolidated net sales in fiscal 2022 compared to fiscal 2021 was primarily attributable to the translation impact of a stronger U.S. dollar, a decline in sales from new software releases and video game accessories, partially offset by an increase in sales of new gaming hardware and toys and collectibles. - We receive payment from the delivery partner subsequent to the transfer of food and the payment terms are short-term in nature. - Net cash used in investing activities was $30.0 million in the year ended December 31, 2022, and increased to $73.3 million in the year ended December 31, 2023. - source_sentence: What informs the ESG disclosures mentioned in the text? sentences: - Common Equity Tier 1 (CET1) Capital refers to the total of common stock and related surplus minus treasury stock, retained earnings, AOCI, and qualifying minority interests after factoring in the necessary regulatory adjustments and deductions. - Constant currency revenue percentage change is calculated by determining the change in current period revenues over prior period revenues where current period foreign currency revenues are translated using prior year exchange outstanding rates and hedging effects are excluded from revenues of both periods. - Our ESG disclosures are also informed by relevant topics identified through third-party ESG reporting organizations, frameworks and standards, such as the TCFD. - source_sentence: How many new aircraft did Delta Air Lines take delivery of in 2023? sentences: - In 2023, Delta took delivery of 43 aircraft. - The listing of our common stock on the NYSE could potentially create a conflict between the exchange’s regulatory responsibilities to vigorously oversee the listing and trading of securities, on the one hand, and our commercial and economic interest, on the other hand. - 'The Company''s enterprise DEI Strategy is aligned to the DEI Vision and Mission and rests on four core pillars: •Build a workforce of individuals with diverse backgrounds, cultures, abilities and perspectives •Foster a culture of inclusion where every individual belongs •Transform talent and business processes to achieve equitable opportunities for all •Drive innovation and growth with our business to serve diverse markets around the world.' pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 model-index: - name: BGE base Financial Matryoshka results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 768 type: dim_768 metrics: - type: cosine_accuracy@1 value: 0.7 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8328571428571429 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8614285714285714 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9171428571428571 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.7 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2776190476190476 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17228571428571426 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09171428571428569 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.7 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8328571428571429 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8614285714285714 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9171428571428571 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.8082439242024833 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7734971655328796 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7770743874539329 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 512 type: dim_512 metrics: - type: cosine_accuracy@1 value: 0.6914285714285714 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8328571428571429 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8685714285714285 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9185714285714286 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6914285714285714 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2776190476190476 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.1737142857142857 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09185714285714283 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6914285714285714 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8328571428571429 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8685714285714285 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9185714285714286 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.8056533729911755 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7695113378684802 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7731633620598676 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 256 type: dim_256 metrics: - type: cosine_accuracy@1 value: 0.6928571428571428 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8328571428571429 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.87 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.91 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6928571428571428 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2776190476190476 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.174 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09099999999999998 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6928571428571428 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8328571428571429 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.87 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.91 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.8031697277454632 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7687063492063488 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.772758974076829 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 128 type: dim_128 metrics: - type: cosine_accuracy@1 value: 0.67 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8028571428571428 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8628571428571429 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9057142857142857 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.67 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2676190476190476 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17257142857142854 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.09057142857142855 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.67 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8028571428571428 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8628571428571429 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9057142857142857 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7882417708737697 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7505816326530609 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7545140112362249 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 64 type: dim_64 metrics: - type: cosine_accuracy@1 value: 0.6557142857142857 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.7871428571428571 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8171428571428572 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.8742857142857143 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6557142857142857 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2623809523809524 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.16342857142857142 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.08742857142857141 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6557142857142857 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.7871428571428571 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8171428571428572 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.8742857142857143 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7637005971170125 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7285300453514736 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7336775414052045 name: Cosine Map@100 --- # BGE base Financial Matryoshka This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - json - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("ChristianBernhard/bge-base-financial-matryoshka") # Run inference sentences = [ 'How many new aircraft did Delta Air Lines take delivery of in 2023?', 'In 2023, Delta took delivery of 43 aircraft.', 'The listing of our common stock on the NYSE could potentially create a conflict between the exchange’s regulatory responsibilities to vigorously oversee the listing and trading of securities, on the one hand, and our commercial and economic interest, on the other hand.', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Datasets: `dim_768`, `dim_512`, `dim_256`, `dim_128` and `dim_64` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | dim_768 | dim_512 | dim_256 | dim_128 | dim_64 | |:--------------------|:-----------|:-----------|:-----------|:-----------|:-----------| | cosine_accuracy@1 | 0.7 | 0.6914 | 0.6929 | 0.67 | 0.6557 | | cosine_accuracy@3 | 0.8329 | 0.8329 | 0.8329 | 0.8029 | 0.7871 | | cosine_accuracy@5 | 0.8614 | 0.8686 | 0.87 | 0.8629 | 0.8171 | | cosine_accuracy@10 | 0.9171 | 0.9186 | 0.91 | 0.9057 | 0.8743 | | cosine_precision@1 | 0.7 | 0.6914 | 0.6929 | 0.67 | 0.6557 | | cosine_precision@3 | 0.2776 | 0.2776 | 0.2776 | 0.2676 | 0.2624 | | cosine_precision@5 | 0.1723 | 0.1737 | 0.174 | 0.1726 | 0.1634 | | cosine_precision@10 | 0.0917 | 0.0919 | 0.091 | 0.0906 | 0.0874 | | cosine_recall@1 | 0.7 | 0.6914 | 0.6929 | 0.67 | 0.6557 | | cosine_recall@3 | 0.8329 | 0.8329 | 0.8329 | 0.8029 | 0.7871 | | cosine_recall@5 | 0.8614 | 0.8686 | 0.87 | 0.8629 | 0.8171 | | cosine_recall@10 | 0.9171 | 0.9186 | 0.91 | 0.9057 | 0.8743 | | **cosine_ndcg@10** | **0.8082** | **0.8057** | **0.8032** | **0.7882** | **0.7637** | | cosine_mrr@10 | 0.7735 | 0.7695 | 0.7687 | 0.7506 | 0.7285 | | cosine_map@100 | 0.7771 | 0.7732 | 0.7728 | 0.7545 | 0.7337 | ## Training Details ### Training Dataset #### json * Dataset: json * Size: 6,300 training samples * Columns: anchor and positive * Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | anchor | positive | |:-------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | What challenges did the company face in its supply chain during fiscal 2021? | During fiscal 2021, we experienced significant disruptions in our supply chain which impacted our ability to ship products from overseas on a timely basis. | | Is the information on Legal proceedings in the report straightforward or referenced to another section? | The information on Legal proceedings called for by Item 3 is incorporated by reference to Note 19 of the Notes to Consolidated Financial Statements in Item 8 of the report. | | What factors particularly influence sales comparisons and comparable sales growth according to the annual report? | Sales comparisons can also be particularly influenced by certain factors that are beyond our control: fluctuations in currency exchange rates (with respect to our international operations); inflation or deflation and changes in the cost of gasoline and associated competitive conditions. | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `gradient_accumulation_steps`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 4 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `bf16`: True - `tf32`: True - `load_best_model_at_end`: True - `optim`: adamw_torch_fused - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 16 - `eval_accumulation_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 4 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: True - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | dim_768_cosine_ndcg@10 | dim_512_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 | |:----------:|:------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:| | 0.8122 | 10 | 1.5819 | - | - | - | - | - | | 0.9746 | 12 | - | 0.7909 | 0.7912 | 0.7907 | 0.7723 | 0.7444 | | 1.6244 | 20 | 0.6676 | - | - | - | - | - | | 1.9492 | 24 | - | 0.7991 | 0.7994 | 0.7983 | 0.7849 | 0.7571 | | 2.4365 | 30 | 0.4321 | - | - | - | - | - | | 2.9239 | 36 | - | 0.8089 | 0.8048 | 0.8016 | 0.7879 | 0.7637 | | 3.2487 | 40 | 0.3958 | - | - | - | - | - | | **3.8985** | **48** | **-** | **0.8082** | **0.8057** | **0.8032** | **0.7882** | **0.7637** | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.12 - Sentence Transformers: 3.3.1 - Transformers: 4.41.2 - PyTorch: 2.1.2+cu121 - Accelerate: 1.2.0 - Datasets: 2.19.1 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```