metadata
library_name: transformers
base_model:
- AI-Sweden-Models/gpt-sw3-6.7b-v2
datasets:
- skvarre/sv-instruct-v1
Model Details
Finetune of gpt-sw3-6.7b-v2 using LoRA with 4-bit quantization. Adapters merged with base model, however with bfloat16 precision tensors.
Usage:
This is a finetune experiment. How to use will be provided later.
Evaluation
ScandEval benchmarks:
| Dataset | Performance (Metric 1 / Metric 2) |
|---|---|
| swerec | 74.95 ± 1.17 / 61.38 ± 1.37 |
| suc3 | 30.75 ± 4.11 / 25.69 ± 4.83 |
| scala-sv | 8.96 ± 2.09 / 51.50 ± 2.94 |
| scandiqa-sv | 50.71 ± 0.99 / 56.76 ± 0.89 |
| swedn | 64.37 ± 0.72 / 18.25 ± 0.29 |
| mmlu-sv | 5.45 ± 0.91 / 28.14 ± 0.82 |
| hellaswag-sv | 27.95 ± 0.73 / 4.19 ± 0.94 |
| speed | 5322.20 ± 1132.75 / 1280.06 ± 408.08 |