metadata

base_model: mistralai/Mistral-Nemo-Instruct-2407
library_name: peft
license: other
tags:
  - llama-factory
  - lora
  - generated_from_trainer
model-index:
  - name: combined_sft_10000_mcq_1epoch
    results: []

combined_sft_10000_mcq_1epoch

This model is a fine-tuned version of mistralai/Mistral-Nemo-Instruct-2407 on the combined_10000_mcq dataset. It achieves the following results on the evaluation set:

Loss: 0.0013

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 10
eval_batch_size: 10
seed: 42
distributed_type: multi-GPU
num_devices: 2
total_train_batch_size: 20
total_eval_batch_size: 20
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
0.0043	0.0333	30	0.0045
0.0041	0.0667	60	0.0040
0.0042	0.1	90	0.0039
0.0038	0.1333	120	0.0038
0.0036	0.1667	150	0.0037
0.0038	0.2	180	0.0037
0.0039	0.2333	210	0.0038
0.0038	0.2667	240	0.0037
0.0034	0.3	270	0.0031
0.0032	0.3333	300	0.0026
0.0027	0.3667	330	0.0025
0.0022	0.4	360	0.0024
0.002	0.4333	390	0.0022
0.0025	0.4667	420	0.0022
0.0023	0.5	450	0.0021
0.0015	0.5333	480	0.0018
0.0017	0.5667	510	0.0017
0.0024	0.6	540	0.0020
0.0019	0.6333	570	0.0018
0.0015	0.6667	600	0.0016
0.0018	0.7	630	0.0015
0.0014	0.7333	660	0.0015
0.0015	0.7667	690	0.0015
0.0013	0.8	720	0.0014
0.0014	0.8333	750	0.0014
0.0017	0.8667	780	0.0014
0.0016	0.9	810	0.0013
0.0017	0.9333	840	0.0013
0.0011	0.9667	870	0.0013
0.0015	1.0	900	0.0013

Framework versions

PEFT 0.12.0
Transformers 4.46.0
Pytorch 2.4.0+cu121
Datasets 2.21.0
Tokenizers 0.20.1