Model Card for educa-ai-nemo-dpo
Model Details
Model Description
educa-ai-nemo-dpo is the preference-aligned version of our SFT model DigitalLearningGmbH/educa-ai-nemo-sft,
using our internal dataset which contains a unique mix of German and English preference data covering a multitude of domains.
In its creation we have paid special attention to data points that can improve performance in German, especially the educational field (text analysis, supporting students in completing textual tasks, ...).
This is a preliminary release and subject to changes or updates.
- Developed by: Digital Learning GmbH
- Funded by [optional]: Digital Learning GmbH
- Shared by [optional]: Digital Learning GmbH
- Model type: Transformer Decoder LLM
- Language(s) (NLP): English, French, German, Spanish, Italian, Portuguese, Russian, Chinese, Japanese
- License: Apache License 2.0
- Finetuned from model: DigitalLearningGmbH/educa-ai-nemo-sft
Uses
As stated before, this is a preliminary release and we are still benchmarking the model as well as improving our datasets for possible further training. As such, we do not recommend using this model in a production setting yet and are looking forward to engaging with the community regarding possible downstream uses and improvements.
Bias, Risks, and Limitations
Refer to the original model card for an overview of the general risks associated with using this model. As this version is only fine-tuned using SFT without any preference alignment, the model may output harmful data. Use is at your own discretion, taking into account the potential risks.
How to Get Started with the Model
Refer to the original model card for code examples.
Be aware that this model uses a slightly different chat template from the original: system prompts are placed before the first user prompt (before the first instance of [INST]).
We include the updated template in the tokenizer config, so you can use tokenizer.apply_chat_template.
Training Details
Instead of standard sigmoid DPO Loss, we used DPO-Positive as we found it improved training stability and overall performance with our dataset.
Training Data
The model has been trained on a mix of some publically-available and permissively-licensed data as well as a majority of unique internal datasets which we have created. Our data encompasses examples of a length up to 16384 tokens, further enhancing the model's long-context capability.
Evaluation
We ran all benchmarks using lm-eval with --apply_chat_template.
For comparison, we performed the same benchmarks on the base model and Llama-3.1-8B-Instruct as well, in the exact same environment with the same parameters.
English Benchmarks
| Benchmark | Llama-3.1-8B-Instruct | Mistral-Nemo-Instruct-2407 | educa-ai-nemo-dpo |
|---|---|---|---|
| hellaswag (acc_norm) | 72.6% | 71.9% | 77.6% |
| winogrande (acc) | 68.0% | 69.8% | 75.2% |
| openbookqa (acc_norm) | 49.0% | 45.8% | 47.0% |
| commonsense_qa (acc) | 64.9% | 74.4% | 75.4% |
| truthfulqa_mc1 (acc) | 40.4% | 39.66% | 41.5% |
| mmlu (acc) | 63.2% | 64.9% | 66.5% |
| triviaqa (exact_match) | 5.3% | 12.3% | 23.99% |
| agieval (acc) | 36.3% | 36.6% | 39.1% |
| arc_challenge (acc_norm) | 54.1% | 52.5% | 54.4% |
| arc_easy (acc_norm) | 75.7% | 74.1% | 76.0% |
| piqa (acc_norm) | 79.6% | 78.9% | 81.5% |
| leaderboard_bbh (acc_norm) | 37.4% | 49.1% | 53.0% |
| leaderboard_gpqa (acc_norm) | 28.5% | 30.6% | 29.4% |
| leaderboard_ifeval (inst_level_loose_acc) | 84.7% | 72.8% | 75.1% |
| leaderboard_mmlu_pro (acc) | 16.2% | 35.1% | 33.67% |
| leaderboard_musr (acc_norm) | 38.8% | 39.3% | 40.2% |
Multilingual Benchmarks
| Benchmark | Llama-3.1-8B-Instruct | Mistral-Nemo-Instruct-2407 | educa-ai-nemo-dpo |
|---|---|---|---|
| global_mmlu_full (acc) | |||
| - de | 48.2% | 55.8% | 57.5% |
| - en | 60.0% | 63.1% | 63.8% |
| - es | 54.7% | 58.1% | 58.9% |
| - fr | 48.3% | 56.3% | 58.1% |
| - it | 51.0% | 58.1% | 59.6% |
| - ja | 47.4% | 50.0% | 51.0% |
| - pt | 23.0% | 43.5% | 55.7% |
| - ru | 41.4% | 54.9% | 55.0% |
| - zh | 49.7% | 52.2% | 55.6% |
| arc_challenge_mt (acc_norm) | |||
| - de | 39.9% | 42.6% | 46.8% |
| - es | 42.8% | 45.6% | 47.3% |
| - it | 43.9% | 44.3% | 46.7% |
| - pt | 41.9% | 42.3% | 46.8% |
| xnli (acc) | |||
| - de | 48.1% | 47.6% | 47.1% |
| - en | 52.4% | 57.3% | 57.8% |
| - es | 46.3% | 45.0% | 47.0% |
| - fr | 51.6% | 38.5% | 40.0% |
| - ru | 48.1% | 41.8% | 38.6% |
| - zh | 40.3% | 36.3% | 36.1% |
| xquad (f1) | |||
| - de | 30.4% | 22.7% | 35.6% |
| - en | 35.0% | 21.8% | 29.9% |
| - es | 31.2% | 17.6% | 29.6% |
| - ru | 39.6% | 24.6% | 37.3% |
| - zh | 28.8% | 10.0% | 16.7% |
Model Card Authors [optional]
This model card was written by Lennard Michael Strohmeyer
Model Card Contact
- Downloads last month
- 28
Model tree for DigitalLearningGmbH/educa-ai-nemo-dpo
Base model
mistralai/Mistral-Nemo-Base-2407