Hugging Face's logo

imdatta0
/
Meta-Llama-3-8B_pct_ortho

PEFT
Safetensors
unsloth
Generated from Trainer
Model card Files Community
Meta-Llama-3-8B_pct_ortho / README.md
imdatta0's picture
imdatta0
End of training
f36f2aa verified
preview code
|
raw
history blame
3.91 kB
metadata
base_model: unsloth/llama-3-8b-bnb-4bit
library_name: peft
license: llama3
tags:
  - unsloth
  - generated_from_trainer
model-index:
  - name: Meta-Llama-3-8B_pct_ortho
    results: []

Meta-Llama-3-8B_pct_ortho

This model is a fine-tuned version of unsloth/llama-3-8b-bnb-4bit on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 2.2409

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0003
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 64
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.02
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss
2.3045 0.0206 8 2.2825
2.2746 0.0412 16 2.2479
2.2264 0.0618 24 2.2587
2.3001 0.0824 32 2.2813
2.3006 0.1030 40 2.2798
2.2719 0.1236 48 2.2780
2.2942 0.1442 56 2.2881
2.314 0.1648 64 2.2993
2.2747 0.1854 72 2.3101
2.3086 0.2060 80 2.3069
2.3318 0.2266 88 2.2899
2.3957 0.2472 96 2.3000
2.3704 0.2678 104 2.2998
2.3319 0.2884 112 2.3124
2.3908 0.3090 120 2.3099
2.3865 0.3296 128 2.3063
2.3306 0.3502 136 2.2947
2.326 0.3708 144 2.2973
2.3421 0.3914 152 2.2987
2.3277 0.4120 160 2.2820
2.3739 0.4326 168 2.2931
2.3157 0.4532 176 2.2898
2.3296 0.4738 184 2.2915
2.3274 0.4944 192 2.2818
2.3225 0.5150 200 2.2861
2.3181 0.5356 208 2.2817
2.3393 0.5562 216 2.2708
2.3276 0.5768 224 2.2763
2.3053 0.5974 232 2.2791
2.2739 0.6180 240 2.2721
2.311 0.6386 248 2.2749
2.3049 0.6592 256 2.2706
2.2615 0.6798 264 2.2703
2.312 0.7004 272 2.2633
2.3017 0.7210 280 2.2594
2.3066 0.7416 288 2.2572
2.2966 0.7621 296 2.2579
2.3375 0.7827 304 2.2461
2.2704 0.8033 312 2.2474
2.2512 0.8239 320 2.2496
2.2834 0.8445 328 2.2431
2.2962 0.8651 336 2.2452
2.3071 0.8857 344 2.2405
2.2739 0.9063 352 2.2401
2.2437 0.9269 360 2.2435
2.2634 0.9475 368 2.2417
2.3116 0.9681 376 2.2406
2.2995 0.9887 384 2.2409

Framework versions

  • PEFT 0.12.0
  • Transformers 4.44.0
  • Pytorch 2.4.0+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1