sizhkhy/reliance

llm3br256

This model is a fine-tuned version of meta-llama/Llama-3.2-3B-Instruct on the reliance dataset. It achieves the following results on the evaluation set:

• Loss: 0.0014

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 25.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.0197	0.2024	25	0.0177
0.0118	0.4049	50	0.0114
0.0103	0.6073	75	0.0080
0.0091	0.8097	100	0.0072
0.003	1.0121	125	0.0060
0.0036	1.2146	150	0.0053
0.0052	1.4170	175	0.0049
0.0029	1.6194	200	0.0042
0.0017	1.8219	225	0.0039
0.0022	2.0243	250	0.0035
0.0027	2.2267	275	0.0032
0.0017	2.4291	300	0.0030
0.0012	2.6316	325	0.0028
0.002	2.8340	350	0.0025
0.0008	3.0364	375	0.0026
0.0012	3.2389	400	0.0025
0.0013	3.4413	425	0.0021
0.0008	3.6437	450	0.0019
0.0014	3.8462	475	0.0020
0.0009	4.0486	500	0.0020
0.0006	4.2510	525	0.0019
0.0005	4.4534	550	0.0018
0.0007	4.6559	575	0.0017
0.0004	4.8583	600	0.0016
0.0004	5.0607	625	0.0016
0.0004	5.2632	650	0.0015
0.0004	5.4656	675	0.0016
0.0005	5.6680	700	0.0015
0.0003	5.8704	725	0.0015
0.0006	6.0729	750	0.0016
0.0003	6.2753	775	0.0015
0.0003	6.4777	800	0.0015
0.0003	6.6802	825	0.0015
0.0004	6.8826	850	0.0014
0.0003	7.0850	875	0.0015
0.0004	7.2874	900	0.0014
0.0004	7.4899	925	0.0015
0.0004	7.6923	950	0.0014
0.0005	7.8947	975	0.0014
0.0003	8.0972	1000	0.0014
0.0003	8.2996	1025	0.0014
0.0003	8.5020	1050	0.0016
0.0003	8.7045	1075	0.0014
0.0003	8.9069	1100	0.0015
0.0003	9.1093	1125	0.0014
0.0004	9.3117	1150	0.0016
0.0002	9.5142	1175	0.0014
0.0005	9.7166	1200	0.0014
0.0002	9.9190	1225	0.0014
0.0002	10.1215	1250	0.0014
0.0002	10.3239	1275	0.0014
0.0002	10.5263	1300	0.0014
0.0002	10.7287	1325	0.0015

Framework versions

• PEFT 0.12.0
• Transformers 4.46.1
• Pytorch 2.4.0+cu121
• Datasets 3.1.0
• Tokenizers 0.20.3