KoModernBERT-base-mlm-v01-ckp01

This model is a fine-tuned version of x2bee/KoModernBERT-base-mlm-v01 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.7654

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 8
• total_train_batch_size: 512
• total_eval_batch_size: 64
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
25.1808	0.1051	1000	3.1264
24.5518	0.2102	2000	3.0455
23.9605	0.3153	3000	nan
23.8549	0.4204	4000	2.9420
23.2944	0.5255	5000	2.9115
23.0988	0.6306	6000	2.8620
22.7786	0.7357	7000	2.8250
22.4325	0.8408	8000	2.7785
21.9527	0.9459	9000	2.7654

Framework versions

• Transformers 4.48.0.dev0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0