README.md

---
language:
- en
license: llama2
tags:
- generated_from_trainer
- finance
model-index:
- name: completed-model
  results:
  - task:
      type: text-generation
    dataset:
      name: ai2_arc
      type: ai2_arc
    metrics:
    - type: AI2 Reasoning Challenge (25-Shot)
      value: 71.93
      name: AI2 Reasoning Challenge (25-Shot)
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      name: Open LLM Leaderboard
  - task:
      type: text-generation
    dataset:
      name: hellaswag
      type: hellaswag
    metrics:
    - type: HellaSwag (10-shot)
      value: 86.82
      name: HellaSwag (10-shot)
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      name: Open LLM Leaderboard
  - task:
      type: text-generation
    dataset:
      name: multiple
      type: miltiple
    metrics:
    - type: MMLU (5-shot)
      value: 70.38
      name: MMLU (5-shot)
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      name: Open LLM Leaderboard
  - task:
      type: text-generation
    dataset:
      name: truthful_qa
      type: truthful_qa
    metrics:
    - type: TruthfulQA (0-shot)
      value: 65.21
      name: TruthfulQA (0-shot)
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      name: Open LLM Leaderboard
  - task:
      type: text-generation
    dataset:
      name: winogrande
      type: winogrande
    metrics:
    - type: Winogrande (5-shot)
      value: 83.58
      name: Winogrande (5-shot)
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      name: Open LLM Leaderboard
  - task:
      type: text-generation
    dataset:
      name: gsm8k
      type: gsm8k
    metrics:
    - type: GSM8k (5-shot)
      value: 61.79
      name: GSM8k (5-shot)
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: AI2 Reasoning Challenge (25-Shot)
      type: ai2_arc
      config: ARC-Challenge
      split: test
      args:
        num_few_shot: 25
    metrics:
    - type: acc_norm
      value: 71.93
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=gradientai/v-alpha-tross
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HellaSwag (10-Shot)
      type: hellaswag
      split: validation
      args:
        num_few_shot: 10
    metrics:
    - type: acc_norm
      value: 86.82
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=gradientai/v-alpha-tross
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU (5-Shot)
      type: cais/mmlu
      config: all
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 70.38
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=gradientai/v-alpha-tross
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: TruthfulQA (0-shot)
      type: truthful_qa
      config: multiple_choice
      split: validation
      args:
        num_few_shot: 0
    metrics:
    - type: mc2
      value: 65.21
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=gradientai/v-alpha-tross
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: Winogrande (5-shot)
      type: winogrande
      config: winogrande_xl
      split: validation
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 83.58
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=gradientai/v-alpha-tross
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GSM8k (5-shot)
      type: gsm8k
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 61.79
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=gradientai/v-alpha-tross
      name: Open LLM Leaderboard
---

**Albatross** is a collection of domain-specific language models for finance applications developed by [Gradient](https://gradient.ai/).

This is the repository for an early, limited capability version, the `v-alpha-tross`, designed to showcase performance on

- mathematical reasoning
- tabular understanding
- open-book retrieval (RAG) & summarization
- conversational interface

Release versions of Albatross models are additionally trained on proprietary implementations of the latest architecture augmentation, expanded training and alignment data, and target reduced hallucination at retrieval, improved auditability, and multi-hop reasoning. To inquire for access to release versions, please reach out to [[email protected]](mailto:[email protected])

## Model description

The `v-alpha-tross` model is based on [meta-llama/Llama-2-70b-hf](https://huggingface.co/meta-llama/Llama-2-70b-hf), with additional, finance specific, pre-training, fine-tuning and instruction tuning.

This model substantially outperforms Llama2-70B models on H6 Average score, and GSM8K, with similar performance to [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1). It also reaches `gpt-3.5-turbo` performance in extracting information from tabular data like those found in SEC filings.

## Intended use

The `v-alpha-tross` is intended as a demonstration of Gradient’s Albatross framework for developing large language models specific to the finance domain. We welcome additional research and development, but do not plan on continued internal development on this legacy model.

To get the expected performance, follow formatting requirements of *Llama-2 chat*, including `INST` and `<<SYS>>` tags, and `<s>` tokens.

## Training Strategy

The Albatross framework overcomes deficiencies in general-purpose language models that arise in the face of solving tasks in the finance domain.

Release versions of Albatross use an expanded data universe for extended capabilities.

### Pre-Training

A base Llama2-70B is further pre-trained on finance data since LLMs are poor at answering questions when their internal relevant document store is sparse [1].

To curate quality training data with low operational overhead we demo a novel data gathering approach:

1. Crawl public repositories of text data. For `v-alpha-tross`, we limited to [Red Pajamas](https://github.com/togethercomputer/RedPajama-Data) and https://github.com/.
2. Programmatically filter the crawled corpus to datasets not likely to be in the base model's training already, using a likelihood ratio test adapted from LiRA membership inference.[2]
3. Human finance professionals review the (much smaller) filtered corpus to further remove low quality results.

[1] Kandpal, Nikhil, et al. "Large language models struggle to learn long-tail knowledge." International Conference on Machine Learning. PMLR, 2023.

[2] Carlini, Nicholas, et al. "Membership inference attacks from first principles." 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.

### Fine-Tuning

Supervised fine-tuning (SFT) and direct preference optimization (DPO)[3] further enhances performance and alignment on finance-related tasks.

`v-alpha-tross` includes a subset of Albatross tuning goals: financial anchoring, mathematical reasoning, tabular understanding, conversational communication, summarization.

| Category | # Tokens (1Ms) | % of Total |
| --- | --- | --- |
| Chat (e.g. [ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k)) | 640 | 45.2 |
| Alignment * (e.g. [orca_dpo](https://huggingface.co/datasets/Intel/orca_dpo_pairs)) | 331 | 23.4 |
| Math * (e.g. Goat[4]) | 300 | 21.2 |
| Tabular * | 68 | 4.8 |
| Summarization (e.g. [legal_summarization](https://huggingface.co/datasets/lighteval/legal_summarization)) | 52 | 3.7 |
| Open-book (e.g. [selfrag](https://huggingface.co/datasets/selfrag/selfrag_train_data)) | 25 | 1.8 |

(*) = Proprietary or includes proprietary data sets

[3] Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D. and Finn, C., 2023. Direct preference optimization: Your language model is secretly a reward model. NeurIPS.

[4] Liu, Tiedong, and Bryan Kian Hsiang Low. "Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks." arXiv preprint arXiv:2305.14201 (2023).

## Benchmarks

From a Llama-2-70B base, `v-alpha-tross` improves H6 metrics, and in particular GSM8k (arithmetic reasoning), scoring similar to Mixtral-8x7B-Instruct-v0.1. Relative to a subset of Open LLM Leaderboard [4] models which also use Llama-2-70B as a base, the model achieves state of the art results for the Average H6 score.

On financial table understanding (our new metric) the model is on par with GPT-3.5.

| Model | H6 [4] | GSM8k | sec_tables_v1 |
| --- | --- | --- | --- |
| v-alpha-tross | 73.28 | 61.79 | 100.0 |
| meta-llama/Llama-2-70B-hf | 67.87 | 54.06 | 75.76 |
| meta-llama/Llama-2-70b-chat-hf | 62.40 | 26.69 | 87.88 |
| mistralai/Mixtral-8x7B-Instruct-v0.1 | 72.70 | 61.11 | 82.35 |
| GPT-3.5 | N/A | 57.1 [5] | 100.0 |

[4]
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

[5]
https://paperswithcode.com/sota/arithmetic-reasoning-on-gsm8k

## Training procedure

We develop Albatross on Gradient’s distributed training platform, leveraging leading open source toolsets and optimizations like [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl), [Accelerate](https://github.com/huggingface/accelerate), and [Deepspeed](https://github.com/microsoft/DeepSpeed) for high throughput and memory efficiency.

### Training hyperparameters (DPO)

The following hyperparameters were used during DPO training:

- learning_rate: 5e-07
- train_batch_size: 3
- eval_batch_size: 3
- seed: 42
- distributed_type: multi-GPU
- num_devices: 40
- total_train_batch_size: 120
- total_eval_batch_size: 120
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 18
- num_epochs: 1
- dpo_beta: .1

### Framework versions

- Transformers 4.35.1
- Pytorch 2.0.1+cu118
- Datasets 2.14.7
- Tokenizers 0.14.1

## Bias

`v-alpha-tross` has not been specifically aligned for safety, so the model can produce problematic outputs (especially when prompted to do so). It is also subject to any risks of the corpus that was used to train the base Llama 2 models.

## More information & how to cite

Whitepaper coming soon!

## The Gradient AI Team

Gradient is accelerating AI transformation across industries. https://gradient.ai/

## Contact Us

Drop an email to [[email protected]](mailto:[email protected])
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_gradientai__v-alpha-tross)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |73.28|
|AI2 Reasoning Challenge (25-Shot)|71.93|
|HellaSwag (10-Shot)              |86.82|
|MMLU (5-Shot)                    |70.38|
|TruthfulQA (0-shot)              |65.21|
|Winogrande (5-shot)              |83.58|
|GSM8k (5-shot)                   |61.79|