README.md

metadata

language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:6300
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
base_model: BAAI/bge-base-en-v1.5
widget:
  - source_sentence: >-
      Goodwill is recognized for the excess of the purchase price over the fair
      value of tangible and identifiable intangible net assets of businesses
      acquired. In evaluating goodwill impairment, a qualitative assessment is
      performed to determine the likelihood that the fair value of a reporting
      unit is less than its carrying amount. This might lead to further testing
      of goodwill for impairment, which includes comparing the fair value of the
      reporting unit to its carrying value (including attributable goodwill).
      Fair value for our reporting units is determined using an income or market
      approach incorporating market participant considerations and management’s
      assumptions.
    sentences:
      - >-
        How is goodwill reviewed for impairment in a company, and what methods
        are used to determine the fair value of reporting units?
      - >-
        What regulatory framework does the FCC currently apply to broadband
        internet access services as of 2023?
      - >-
        What were the total interest payments made by the company in 2023, 2022,
        and 2021?
  - source_sentence: >-
      Part IV Item 15, titled 'Exhibits, Financial Statement Schedules',
      includes the 'Index to Financial Statements' and the 'Index to Financial
      Statement Schedules.'
    sentences:
      - >-
        What was the net cash from operations reported for the year ended June
        30, 2023?
      - >-
        Where would one find the 'Index to Financial Statements' and the 'Index
        to Financial Statement Schedules' mentioned?
      - What is the trajectory of the AMPTC for microinverters starting in 2030?
  - source_sentence: >-
      As of December 31, 2023, the total amortized cost, net of valuation
      allowance, for non-U.S. government securities amounted to $14,516 million.
    sentences:
      - >-
        How much federal net operating loss carryforwards did the company have
        at the end of 2023, and how much of it is expected to be realized?
      - >-
        What accounting principles are followed in the preparation of Goldman
        Sachs' consolidated financial statements for 2023?
      - >-
        What was the total amortized cost, net of valuation allowance, for
        non-U.S. government securities as of December 31, 2023?
  - source_sentence: >-
      Information about legal proceedings in the Annual Report on Form 10-K is
      incorporated by reference under several notes and sections.
    sentences:
      - >-
        What method is used to provide information about legal proceedings in
        the Annual Report on Form 10-K?
      - What can cause delays in pharmaceutical product launches?
      - >-
        What was the total amount of cash dividends declared by the company per
        share in the fiscal year ending on October 1, 2023?
  - source_sentence: >-
      MERS database revenues contain multiple performance obligations related to
      each new loan registration and future transfers, and the revenues are
      primarily recorded at the point in time of each transaction.
    sentences:
      - How are revenues from MERS database recognized?
      - >-
        How did The Home Depot, Inc.'s basic earnings per share change from 2020
        to 2022?
      - >-
        How many active sellers and buyers did Etsy's marketplaces connect as of
        December 31, 2023?
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
model-index:
  - name: BGE base Financial Matryoshka
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 768
          type: dim_768
        metrics:
          - type: cosine_accuracy@1
            value: 0.7228571428571429
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8528571428571429
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.89
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9185714285714286
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7228571428571429
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2842857142857143
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.17799999999999996
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09185714285714283
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7228571428571429
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8528571428571429
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.89
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9185714285714286
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8244010006831627
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7936836734693877
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7971656786986449
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 512
          type: dim_512
        metrics:
          - type: cosine_accuracy@1
            value: 0.7157142857142857
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8471428571428572
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8857142857142857
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9185714285714286
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7157142857142857
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.28238095238095234
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.17714285714285713
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09185714285714283
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7157142857142857
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8471428571428572
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8857142857142857
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9185714285714286
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8209116379330612
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7891343537414967
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7926472335071902
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 256
          type: dim_256
        metrics:
          - type: cosine_accuracy@1
            value: 0.71
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8428571428571429
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8757142857142857
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9128571428571428
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.71
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.28095238095238095
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.17514285714285713
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09128571428571428
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.71
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8428571428571429
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8757142857142857
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9128571428571428
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8138965576076403
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7818429705215417
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7855894139852542
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 128
          type: dim_128
        metrics:
          - type: cosine_accuracy@1
            value: 0.6871428571428572
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8185714285714286
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8628571428571429
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8971428571428571
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.6871428571428572
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.27285714285714285
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.17257142857142854
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.0897142857142857
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.6871428571428572
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8185714285714286
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8628571428571429
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.8971428571428571
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7953389524625682
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7622392290249432
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7667451557566504
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 64
          type: dim_64
        metrics:
          - type: cosine_accuracy@1
            value: 0.6628571428571428
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.7842857142857143
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8371428571428572
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8771428571428571
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.6628571428571428
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.26142857142857145
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.1674285714285714
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.0877142857142857
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.6628571428571428
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.7842857142857143
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8371428571428572
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.8771428571428571
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7701231991584621
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7357777777777779
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7410692697767751
            name: Cosine Map@100

BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5 on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("chcho/bge-base-financial-matryoshka")
# Run inference
sentences = [
    'MERS database revenues contain multiple performance obligations related to each new loan registration and future transfers, and the revenues are primarily recorded at the point in time of each transaction.',
    'How are revenues from MERS database recognized?',
    "How many active sellers and buyers did Etsy's marketplaces connect as of December 31, 2023?",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: dim_768, dim_512, dim_256, dim_128 and dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	dim_768	dim_512	dim_256	dim_128	dim_64
cosine_accuracy@1	0.7229	0.7157	0.71	0.6871	0.6629
cosine_accuracy@3	0.8529	0.8471	0.8429	0.8186	0.7843
cosine_accuracy@5	0.89	0.8857	0.8757	0.8629	0.8371
cosine_accuracy@10	0.9186	0.9186	0.9129	0.8971	0.8771
cosine_precision@1	0.7229	0.7157	0.71	0.6871	0.6629
cosine_precision@3	0.2843	0.2824	0.281	0.2729	0.2614
cosine_precision@5	0.178	0.1771	0.1751	0.1726	0.1674
cosine_precision@10	0.0919	0.0919	0.0913	0.0897	0.0877
cosine_recall@1	0.7229	0.7157	0.71	0.6871	0.6629
cosine_recall@3	0.8529	0.8471	0.8429	0.8186	0.7843
cosine_recall@5	0.89	0.8857	0.8757	0.8629	0.8371
cosine_recall@10	0.9186	0.9186	0.9129	0.8971	0.8771
cosine_ndcg@10	0.8244	0.8209	0.8139	0.7953	0.7701
cosine_mrr@10	0.7937	0.7891	0.7818	0.7622	0.7358
cosine_map@100	0.7972	0.7926	0.7856	0.7667	0.7411

Training Details

Training Dataset

json

• Dataset: json
• Size: 6,300 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 2 tokens mean: 46.48 tokens max: 512 tokens	min: 2 tokens mean: 20.5 tokens max: 51 tokens

• Samples:

  positive	anchor
  We use a variety of methodologies to determine the fair value of these assets, including discounted cash flow models, which include assumptions we believe are consistent with those a market participant would use.	How is the fair value of intangible assets determined within a company?
  We continue to own a 35% minority ownership in Gentiva Hospice operations after it was restructured into a new stand-alone company.	What percentage minority ownership does the company retain in Gentiva Hospice after the restructuring?
  The net interest income for the first quarter of 2023 was $14,448 million.	What was the net interest income for the first quarter of 2023?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.8122	10	1.5791	-	-	-	-	-
0.9746	12	-	0.8089	0.8028	0.7958	0.7714	0.7428
1.6244	20	0.6637	-	-	-	-	-
1.9492	24	-	0.8209	0.8166	0.8109	0.7913	0.7615
2.4365	30	0.5072	-	-	-	-	-
2.9239	36	-	0.8229	0.82	0.8133	0.7959	0.7704
3.2487	40	0.394	-	-	-	-	-
3.8985	48	-	0.8244	0.8209	0.8139	0.7953	0.7701

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.9.5
• Sentence Transformers: 3.3.1
• Transformers: 4.41.2
• PyTorch: 2.1.2+cu121
• Accelerate: 0.27.2
• Datasets: 2.19.1
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}