Update README.md

2593553 verified 8 months ago

6.59 kB

	---
	language:
	- fi
	base_model: HPLT/gpt-7b-nordic-prerelease
	license: apache-2.0
	datasets:
	- pinzhenchen/alpaca-cleaned-fi
	---

	# Model Card for NordicAlpaca-Finnish-V1-7B

	This is a merge of [mpasila/NordicAlpaca-Finnish-V1-7B-LoRA](https://huggingface.co/mpasila/NordicAlpaca-Finnish-V1-7B-LoRA/).

	Dataset used with the LoRA is [pinzhenchen/alpaca-cleaned-fi](https://huggingface.co/datasets/pinzhenchen/alpaca-cleaned-fi/).

	Base model used: [HPLT/gpt-7b-nordic-prerelease](https://huggingface.co/HPLT/gpt-7b-nordic-prerelease/)

	It uses Alpaca format but with a translated instruction at the start:
	```
	{
	"instruction,output": "Alla on ohje, jossa kuvataan tehtävä. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Response:\n%output%",
	"instruction,input,output": "Alla on ohje, jossa kuvataan tehtävä ja joka on yhdistetty kontekstia lisäävään syötteeseen. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Input:\n%input%\n\n### Response:\n%output%"
	}

	```

	Merged using this [Colab notebook](https://colab.research.google.com/drive/1a76Y21GfPtmVs71Uztlgk2xzPA4_vVjs?usp=sharing). It might not be the best way to merge a quantized LoRA on to a float16 model but I just wanted to quickly do something. You can try merging it better if you want.

	## Evaluation

	\| Model \| Size \| Type \| FIN-bench (score) \|
	\|-------\|------\|------\|-------\|
	\| mpasila/NordicAlpaca-Finnish-V1-7B \| 7B \| Instruct \| 0.3891 \|
	\| [mpasila/Finnish-Alpaca-Small-7B](https://huggingface.co/mpasila/Finnish-Alpaca-Small-7B) \| 7B \| Instruct \| 0.3586 \|
	\| [mpasila/Finnish-Alpaca-Tiny-V2-7B](https://huggingface.co/mpasila/Finnish-Alpaca-Tiny-V2-7B) \| 7B \| Instruct \| 0.4654 \|
	\| [mpasila/Alpacazord-Viking-7B](https://huggingface.co/mpasila/Alpacazord-Viking-7B) \| 7B \| Instruct \| 0.4123 \|
	\| [mpasila/Finnish-Viking-Alpaca-V1-7B](https://huggingface.co/mpasila/Finnish-Viking-Alpaca-V1-7B) \| 7B \| Instruct \| 0.3943 \|
	\| [Finnish-NLP/llama-7b-finnish-instruct-v0.1](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.1) \| 7B \| Instruct \| 0.4365 \|
	\| [Finnish-NLP/llama-7b-finnish-instruct-v0.2](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.2) \| 7B \| Instruct \| 0.3993 \|
	\| [Finnish-NLP/llama-7b-finnish](https://huggingface.co/Finnish-NLP/llama-7b-finnish) \| 7B \| Base \| 0.2350 \|
	\| [LumiOpen/Viking-7B (1000B)](https://huggingface.co/LumiOpen/Viking-7B) \| 7B \| Base \| 0.3721 \|
	\| [HPLT/gpt-7b-nordic-prerelease](https://huggingface.co/HPLT/gpt-7b-nordic-prerelease) \| 7B \| Base \| 0.3169 \|

	[Source](https://docs.google.com/spreadsheets/d/1rqJb9dQVihg-Z1_Ras1L_-wuzPg9xNzpdmM2x5HueeY/edit?usp=sharing)

	#### FIN-bench scores:

	\| Task \|Version\| Metric \|Value \| \|Stderr\|
	\|------------------------------------------------\|------:\|---------------------\|-----:\|---\|-----:\|
	\|bigbench_analogies \| 0\|multiple_choice_grade\|0.5615\|± \|0.0437\|
	\|bigbench_arithmetic_1_digit_addition \| 0\|multiple_choice_grade\|0.5300\|± \|0.0502\|
	\|bigbench_arithmetic_1_digit_division \| 0\|multiple_choice_grade\|0.8261\|± \|0.0808\|
	\|bigbench_arithmetic_1_digit_multiplication \| 0\|multiple_choice_grade\|0.3700\|± \|0.0485\|
	\|bigbench_arithmetic_1_digit_subtraction \| 0\|multiple_choice_grade\|0.6000\|± \|0.0492\|
	\|bigbench_arithmetic_2_digit_addition \| 0\|multiple_choice_grade\|0.0300\|± \|0.0171\|
	\|bigbench_arithmetic_2_digit_division \| 0\|multiple_choice_grade\|0.4700\|± \|0.0502\|
	\|bigbench_arithmetic_2_digit_multiplication \| 0\|multiple_choice_grade\|0.1000\|± \|0.0302\|
	\|bigbench_arithmetic_2_digit_subtraction \| 0\|multiple_choice_grade\|0.1800\|± \|0.0386\|
	\|bigbench_arithmetic_3_digit_addition \| 0\|multiple_choice_grade\|0.3000\|± \|0.0461\|
	\|bigbench_arithmetic_3_digit_division \| 0\|multiple_choice_grade\|0.2000\|± \|0.0402\|
	\|bigbench_arithmetic_3_digit_multiplication \| 0\|multiple_choice_grade\|0.1800\|± \|0.0386\|
	\|bigbench_arithmetic_3_digit_subtraction \| 0\|multiple_choice_grade\|0.2500\|± \|0.0435\|
	\|bigbench_arithmetic_4_digit_addition \| 0\|multiple_choice_grade\|0.4200\|± \|0.0496\|
	\|bigbench_arithmetic_4_digit_division \| 0\|multiple_choice_grade\|0.2400\|± \|0.0429\|
	\|bigbench_arithmetic_4_digit_multiplication \| 0\|multiple_choice_grade\|0.2200\|± \|0.0416\|
	\|bigbench_arithmetic_4_digit_subtraction \| 0\|multiple_choice_grade\|0.4600\|± \|0.0501\|
	\|bigbench_arithmetic_5_digit_addition \| 0\|multiple_choice_grade\|0.5400\|± \|0.0501\|
	\|bigbench_arithmetic_5_digit_division \| 0\|multiple_choice_grade\|0.1100\|± \|0.0314\|
	\|bigbench_arithmetic_5_digit_multiplication \| 0\|multiple_choice_grade\|0.2400\|± \|0.0429\|
	\|bigbench_arithmetic_5_digit_subtraction \| 0\|multiple_choice_grade\|0.5200\|± \|0.0502\|
	\|bigbench_cause_and_effect_one_sentence \| 0\|multiple_choice_grade\|0.6471\|± \|0.0676\|
	\|bigbench_cause_and_effect_one_sentence_no_prompt\| 0\|multiple_choice_grade\|0.8039\|± \|0.0561\|
	\|bigbench_cause_and_effect_two_sentences \| 0\|multiple_choice_grade\|0.3529\|± \|0.0676\|
	\|bigbench_emotions \| 0\|multiple_choice_grade\|0.2938\|± \|0.0361\|
	\|bigbench_empirical_judgments \| 0\|multiple_choice_grade\|0.3333\|± \|0.0476\|
	\|bigbench_general_knowledge \| 0\|multiple_choice_grade\|0.2857\|± \|0.0544\|
	\|bigbench_hhh_alignment_harmless \| 0\|multiple_choice_grade\|0.3448\|± \|0.0630\|
	\|bigbench_hhh_alignment_helpful \| 0\|multiple_choice_grade\|0.3220\|± \|0.0614\|
	\|bigbench_hhh_alignment_honest \| 0\|multiple_choice_grade\|0.3729\|± \|0.0635\|
	\|bigbench_hhh_alignment_other \| 0\|multiple_choice_grade\|0.5581\|± \|0.0766\|
	\|bigbench_intent_recognition \| 0\|multiple_choice_grade\|0.1777\|± \|0.0145\|
	\|bigbench_misconceptions \| 0\|multiple_choice_grade\|0.5373\|± \|0.0432\|
	\|bigbench_paraphrase \| 0\|multiple_choice_grade\|0.4750\|± \|0.0354\|
	\|bigbench_sentence_ambiguity \| 0\|multiple_choice_grade\|0.4333\|± \|0.0645\|
	\|bigbench_similarities_abstraction \| 0\|multiple_choice_grade\|0.7237\|± \|0.0516\|

	### Framework versions

	- PEFT 0.8.2

	---
	language:
	- fi
	base_model: HPLT/gpt-7b-nordic-prerelease
	license: apache-2.0
	datasets:
	- pinzhenchen/alpaca-cleaned-fi
	---

	# Model Card for NordicAlpaca-Finnish-V1-7B

	This is a merge of [mpasila/NordicAlpaca-Finnish-V1-7B-LoRA](https://huggingface.co/mpasila/NordicAlpaca-Finnish-V1-7B-LoRA/).

	Dataset used with the LoRA is [pinzhenchen/alpaca-cleaned-fi](https://huggingface.co/datasets/pinzhenchen/alpaca-cleaned-fi/).

	Base model used: [HPLT/gpt-7b-nordic-prerelease](https://huggingface.co/HPLT/gpt-7b-nordic-prerelease/)

	It uses Alpaca format but with a translated instruction at the start:
	```
	{
	"instruction,output": "Alla on ohje, jossa kuvataan tehtävä. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Response:\n%output%",
	"instruction,input,output": "Alla on ohje, jossa kuvataan tehtävä ja joka on yhdistetty kontekstia lisäävään syötteeseen. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Input:\n%input%\n\n### Response:\n%output%"
	}

	```

	Merged using this [Colab notebook](https://colab.research.google.com/drive/1a76Y21GfPtmVs71Uztlgk2xzPA4_vVjs?usp=sharing). It might not be the best way to merge a quantized LoRA on to a float16 model but I just wanted to quickly do something. You can try merging it better if you want.

	## Evaluation

	\| Model \| Size \| Type \| FIN-bench (score) \|
	\|-------\|------\|------\|-------\|
	\| mpasila/NordicAlpaca-Finnish-V1-7B \| 7B \| Instruct \| 0.3891 \|
	\| [mpasila/Finnish-Alpaca-Small-7B](https://huggingface.co/mpasila/Finnish-Alpaca-Small-7B) \| 7B \| Instruct \| 0.3586 \|
	\| [mpasila/Finnish-Alpaca-Tiny-V2-7B](https://huggingface.co/mpasila/Finnish-Alpaca-Tiny-V2-7B) \| 7B \| Instruct \| 0.4654 \|
	\| [mpasila/Alpacazord-Viking-7B](https://huggingface.co/mpasila/Alpacazord-Viking-7B) \| 7B \| Instruct \| 0.4123 \|
	\| [mpasila/Finnish-Viking-Alpaca-V1-7B](https://huggingface.co/mpasila/Finnish-Viking-Alpaca-V1-7B) \| 7B \| Instruct \| 0.3943 \|
	\| [Finnish-NLP/llama-7b-finnish-instruct-v0.1](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.1) \| 7B \| Instruct \| 0.4365 \|
	\| [Finnish-NLP/llama-7b-finnish-instruct-v0.2](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.2) \| 7B \| Instruct \| 0.3993 \|
	\| [Finnish-NLP/llama-7b-finnish](https://huggingface.co/Finnish-NLP/llama-7b-finnish) \| 7B \| Base \| 0.2350 \|
	\| [LumiOpen/Viking-7B (1000B)](https://huggingface.co/LumiOpen/Viking-7B) \| 7B \| Base \| 0.3721 \|
	\| [HPLT/gpt-7b-nordic-prerelease](https://huggingface.co/HPLT/gpt-7b-nordic-prerelease) \| 7B \| Base \| 0.3169 \|

	[Source](https://docs.google.com/spreadsheets/d/1rqJb9dQVihg-Z1_Ras1L_-wuzPg9xNzpdmM2x5HueeY/edit?usp=sharing)

	#### FIN-bench scores:

	\| Task \|Version\| Metric \|Value \| \|Stderr\|
	\|------------------------------------------------\|------:\|---------------------\|-----:\|---\|-----:\|
	\|bigbench_analogies \| 0\|multiple_choice_grade\|0.5615\|± \|0.0437\|
	\|bigbench_arithmetic_1_digit_addition \| 0\|multiple_choice_grade\|0.5300\|± \|0.0502\|
	\|bigbench_arithmetic_1_digit_division \| 0\|multiple_choice_grade\|0.8261\|± \|0.0808\|
	\|bigbench_arithmetic_1_digit_multiplication \| 0\|multiple_choice_grade\|0.3700\|± \|0.0485\|
	\|bigbench_arithmetic_1_digit_subtraction \| 0\|multiple_choice_grade\|0.6000\|± \|0.0492\|
	\|bigbench_arithmetic_2_digit_addition \| 0\|multiple_choice_grade\|0.0300\|± \|0.0171\|
	\|bigbench_arithmetic_2_digit_division \| 0\|multiple_choice_grade\|0.4700\|± \|0.0502\|
	\|bigbench_arithmetic_2_digit_multiplication \| 0\|multiple_choice_grade\|0.1000\|± \|0.0302\|
	\|bigbench_arithmetic_2_digit_subtraction \| 0\|multiple_choice_grade\|0.1800\|± \|0.0386\|
	\|bigbench_arithmetic_3_digit_addition \| 0\|multiple_choice_grade\|0.3000\|± \|0.0461\|
	\|bigbench_arithmetic_3_digit_division \| 0\|multiple_choice_grade\|0.2000\|± \|0.0402\|
	\|bigbench_arithmetic_3_digit_multiplication \| 0\|multiple_choice_grade\|0.1800\|± \|0.0386\|
	\|bigbench_arithmetic_3_digit_subtraction \| 0\|multiple_choice_grade\|0.2500\|± \|0.0435\|
	\|bigbench_arithmetic_4_digit_addition \| 0\|multiple_choice_grade\|0.4200\|± \|0.0496\|
	\|bigbench_arithmetic_4_digit_division \| 0\|multiple_choice_grade\|0.2400\|± \|0.0429\|
	\|bigbench_arithmetic_4_digit_multiplication \| 0\|multiple_choice_grade\|0.2200\|± \|0.0416\|
	\|bigbench_arithmetic_4_digit_subtraction \| 0\|multiple_choice_grade\|0.4600\|± \|0.0501\|
	\|bigbench_arithmetic_5_digit_addition \| 0\|multiple_choice_grade\|0.5400\|± \|0.0501\|
	\|bigbench_arithmetic_5_digit_division \| 0\|multiple_choice_grade\|0.1100\|± \|0.0314\|
	\|bigbench_arithmetic_5_digit_multiplication \| 0\|multiple_choice_grade\|0.2400\|± \|0.0429\|
	\|bigbench_arithmetic_5_digit_subtraction \| 0\|multiple_choice_grade\|0.5200\|± \|0.0502\|
	\|bigbench_cause_and_effect_one_sentence \| 0\|multiple_choice_grade\|0.6471\|± \|0.0676\|
	\|bigbench_cause_and_effect_one_sentence_no_prompt\| 0\|multiple_choice_grade\|0.8039\|± \|0.0561\|
	\|bigbench_cause_and_effect_two_sentences \| 0\|multiple_choice_grade\|0.3529\|± \|0.0676\|
	\|bigbench_emotions \| 0\|multiple_choice_grade\|0.2938\|± \|0.0361\|
	\|bigbench_empirical_judgments \| 0\|multiple_choice_grade\|0.3333\|± \|0.0476\|
	\|bigbench_general_knowledge \| 0\|multiple_choice_grade\|0.2857\|± \|0.0544\|
	\|bigbench_hhh_alignment_harmless \| 0\|multiple_choice_grade\|0.3448\|± \|0.0630\|
	\|bigbench_hhh_alignment_helpful \| 0\|multiple_choice_grade\|0.3220\|± \|0.0614\|
	\|bigbench_hhh_alignment_honest \| 0\|multiple_choice_grade\|0.3729\|± \|0.0635\|
	\|bigbench_hhh_alignment_other \| 0\|multiple_choice_grade\|0.5581\|± \|0.0766\|
	\|bigbench_intent_recognition \| 0\|multiple_choice_grade\|0.1777\|± \|0.0145\|
	\|bigbench_misconceptions \| 0\|multiple_choice_grade\|0.5373\|± \|0.0432\|
	\|bigbench_paraphrase \| 0\|multiple_choice_grade\|0.4750\|± \|0.0354\|
	\|bigbench_sentence_ambiguity \| 0\|multiple_choice_grade\|0.4333\|± \|0.0645\|
	\|bigbench_similarities_abstraction \| 0\|multiple_choice_grade\|0.7237\|± \|0.0516\|

	### Framework versions

	- PEFT 0.8.2