Hugging Face's logo

AmberYifan
/
Gemma-2-9B-Mistral-7B-v0.1-mix

Text Generation
Transformers
Safetensors
gemma2
Generated from Trainer
trl
dpo
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
AmberYifan commited on
Commit
52dbd70
·
verified ·
1 Parent(s): 49d3b1d

Model save

Browse files
Files changed (5) hide show
  1. README.md +68 -0
  2. all_results.json +9 -0
  3. generation_config.json +8 -0
  4. train_results.json +9 -0
  5. trainer_state.json +375 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: AmberYifan/Gemma-2-9B-sft-ultrachat-safeRLHF
3
+ library_name: transformers
4
+ model_name: Gemma-2-9B-Mistral-7B-v0.1-mix
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - dpo
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for Gemma-2-9B-Mistral-7B-v0.1-mix
13
+
14
+ This model is a fine-tuned version of [AmberYifan/Gemma-2-9B-sft-ultrachat-safeRLHF](https://huggingface.co/AmberYifan/Gemma-2-9B-sft-ultrachat-safeRLHF).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="AmberYifan/Gemma-2-9B-Mistral-7B-v0.1-mix", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/yifanwang/huggingface/runs/t7vxh9vi)
31
+
32
+ This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
33
+
34
+ ### Framework versions
35
+
36
+ - TRL: 0.12.2
37
+ - Transformers: 4.46.3
38
+ - Pytorch: 2.5.1+cu118
39
+ - Datasets: 3.2.0
40
+ - Tokenizers: 0.20.3
41
+
42
+ ## Citations
43
+
44
+ Cite DPO as:
45
+
46
+ ```bibtex
47
+ @inproceedings{rafailov2023direct,
48
+ title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
49
+ author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
50
+ year = 2023,
51
+ booktitle = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
52
+ url = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
53
+ editor = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
54
+ }
55
+ ```
56
+
57
+ Cite TRL as:
58
+
59
+ ```bibtex
60
+ @misc{vonwerra2022trl,
61
+ title = {{TRL: Transformer Reinforcement Learning}},
62
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
63
+ year = 2020,
64
+ journal = {GitHub repository},
65
+ publisher = {GitHub},
66
+ howpublished = {\url{https://github.com/huggingface/trl}}
67
+ }
68
+ ```
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.354867279214203,
5
+ "train_runtime": 2422.2159,
6
+ "train_samples": 2000,
7
+ "train_samples_per_second": 2.477,
8
+ "train_steps_per_second": 0.078
9
+ }
generation_config.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 2,
4
+ "cache_implementation": "hybrid",
5
+ "eos_token_id": 1,
6
+ "pad_token_id": 0,
7
+ "transformers_version": "4.46.3"
8
+ }
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.354867279214203,
5
+ "train_runtime": 2422.2159,
6
+ "train_samples": 2000,
7
+ "train_samples_per_second": 2.477,
8
+ "train_steps_per_second": 0.078
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,375 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 3.0,
5
+ "eval_steps": 500,
6
+ "global_step": 189,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.015873015873015872,
13
+ "grad_norm": 32.49287155827833,
14
+ "learning_rate": 2.6315789473684208e-08,
15
+ "logits/chosen": -2.453125,
16
+ "logits/rejected": -1.4453125,
17
+ "logps/chosen": -35.0,
18
+ "logps/rejected": -71.5,
19
+ "loss": 0.6914,
20
+ "rewards/accuracies": 0.0,
21
+ "rewards/chosen": 0.0,
22
+ "rewards/margins": 0.0,
23
+ "rewards/rejected": 0.0,
24
+ "step": 1
25
+ },
26
+ {
27
+ "epoch": 0.15873015873015872,
28
+ "grad_norm": 35.56440458755336,
29
+ "learning_rate": 2.631578947368421e-07,
30
+ "logits/chosen": -1.890625,
31
+ "logits/rejected": -0.7421875,
32
+ "logps/chosen": -65.5,
33
+ "logps/rejected": -58.75,
34
+ "loss": 0.6871,
35
+ "rewards/accuracies": 0.4583333432674408,
36
+ "rewards/chosen": 0.00341796875,
37
+ "rewards/margins": 0.0299072265625,
38
+ "rewards/rejected": -0.0264892578125,
39
+ "step": 10
40
+ },
41
+ {
42
+ "epoch": 0.31746031746031744,
43
+ "grad_norm": 27.925399110912338,
44
+ "learning_rate": 4.970588235294118e-07,
45
+ "logits/chosen": -1.640625,
46
+ "logits/rejected": -0.69140625,
47
+ "logps/chosen": -52.25,
48
+ "logps/rejected": -74.5,
49
+ "loss": 0.7455,
50
+ "rewards/accuracies": 0.5625,
51
+ "rewards/chosen": 0.00579833984375,
52
+ "rewards/margins": 0.0537109375,
53
+ "rewards/rejected": -0.0478515625,
54
+ "step": 20
55
+ },
56
+ {
57
+ "epoch": 0.47619047619047616,
58
+ "grad_norm": 21.690722291304077,
59
+ "learning_rate": 4.676470588235294e-07,
60
+ "logits/chosen": -0.92578125,
61
+ "logits/rejected": -1.0859375,
62
+ "logps/chosen": -62.75,
63
+ "logps/rejected": -57.75,
64
+ "loss": 0.6262,
65
+ "rewards/accuracies": 0.6875,
66
+ "rewards/chosen": 0.10693359375,
67
+ "rewards/margins": 0.1513671875,
68
+ "rewards/rejected": -0.044189453125,
69
+ "step": 30
70
+ },
71
+ {
72
+ "epoch": 0.6349206349206349,
73
+ "grad_norm": 27.607029623598933,
74
+ "learning_rate": 4.38235294117647e-07,
75
+ "logits/chosen": -1.5390625,
76
+ "logits/rejected": -0.6015625,
77
+ "logps/chosen": -52.0,
78
+ "logps/rejected": -97.5,
79
+ "loss": 0.6431,
80
+ "rewards/accuracies": 0.75,
81
+ "rewards/chosen": 0.08740234375,
82
+ "rewards/margins": 1.046875,
83
+ "rewards/rejected": -0.9609375,
84
+ "step": 40
85
+ },
86
+ {
87
+ "epoch": 0.7936507936507936,
88
+ "grad_norm": 89.33304447908654,
89
+ "learning_rate": 4.0882352941176465e-07,
90
+ "logits/chosen": -1.515625,
91
+ "logits/rejected": -1.0625,
92
+ "logps/chosen": -75.0,
93
+ "logps/rejected": -74.0,
94
+ "loss": 0.742,
95
+ "rewards/accuracies": 0.6625000238418579,
96
+ "rewards/chosen": -0.2197265625,
97
+ "rewards/margins": 0.0859375,
98
+ "rewards/rejected": -0.306640625,
99
+ "step": 50
100
+ },
101
+ {
102
+ "epoch": 0.9523809523809523,
103
+ "grad_norm": 28.864344086826605,
104
+ "learning_rate": 3.7941176470588235e-07,
105
+ "logits/chosen": -1.5,
106
+ "logits/rejected": -0.96875,
107
+ "logps/chosen": -59.75,
108
+ "logps/rejected": -84.0,
109
+ "loss": 0.6597,
110
+ "rewards/accuracies": 0.7124999761581421,
111
+ "rewards/chosen": 0.205078125,
112
+ "rewards/margins": 0.478515625,
113
+ "rewards/rejected": -0.2734375,
114
+ "step": 60
115
+ },
116
+ {
117
+ "epoch": 1.0,
118
+ "eval_logits/chosen": -1.3671875,
119
+ "eval_logits/rejected": -0.71875,
120
+ "eval_logps/chosen": -61.25,
121
+ "eval_logps/rejected": -89.5,
122
+ "eval_loss": 0.684765636920929,
123
+ "eval_rewards/accuracies": 0.625,
124
+ "eval_rewards/chosen": 0.0400390625,
125
+ "eval_rewards/margins": 0.4375,
126
+ "eval_rewards/rejected": -0.3984375,
127
+ "eval_runtime": 18.8729,
128
+ "eval_samples_per_second": 10.597,
129
+ "eval_steps_per_second": 0.371,
130
+ "step": 63
131
+ },
132
+ {
133
+ "epoch": 1.1111111111111112,
134
+ "grad_norm": 20.770261653452945,
135
+ "learning_rate": 3.5e-07,
136
+ "logits/chosen": -1.6171875,
137
+ "logits/rejected": -0.87109375,
138
+ "logps/chosen": -62.25,
139
+ "logps/rejected": -85.0,
140
+ "loss": 0.3381,
141
+ "rewards/accuracies": 0.8999999761581421,
142
+ "rewards/chosen": 0.578125,
143
+ "rewards/margins": 1.953125,
144
+ "rewards/rejected": -1.375,
145
+ "step": 70
146
+ },
147
+ {
148
+ "epoch": 1.2698412698412698,
149
+ "grad_norm": 18.43753108370381,
150
+ "learning_rate": 3.205882352941177e-07,
151
+ "logits/chosen": -1.34375,
152
+ "logits/rejected": -0.84375,
153
+ "logps/chosen": -49.5,
154
+ "logps/rejected": -99.0,
155
+ "loss": 0.3254,
156
+ "rewards/accuracies": 0.925000011920929,
157
+ "rewards/chosen": 0.3671875,
158
+ "rewards/margins": 2.390625,
159
+ "rewards/rejected": -2.015625,
160
+ "step": 80
161
+ },
162
+ {
163
+ "epoch": 1.4285714285714286,
164
+ "grad_norm": 16.839062370433073,
165
+ "learning_rate": 2.911764705882353e-07,
166
+ "logits/chosen": -1.5078125,
167
+ "logits/rejected": -0.78515625,
168
+ "logps/chosen": -57.5,
169
+ "logps/rejected": -72.5,
170
+ "loss": 0.2715,
171
+ "rewards/accuracies": 0.8999999761581421,
172
+ "rewards/chosen": 0.84765625,
173
+ "rewards/margins": 2.03125,
174
+ "rewards/rejected": -1.1875,
175
+ "step": 90
176
+ },
177
+ {
178
+ "epoch": 1.5873015873015874,
179
+ "grad_norm": 16.580044861649466,
180
+ "learning_rate": 2.6176470588235295e-07,
181
+ "logits/chosen": -2.09375,
182
+ "logits/rejected": -1.3359375,
183
+ "logps/chosen": -51.0,
184
+ "logps/rejected": -99.5,
185
+ "loss": 0.2519,
186
+ "rewards/accuracies": 0.8374999761581421,
187
+ "rewards/chosen": 0.345703125,
188
+ "rewards/margins": 1.9375,
189
+ "rewards/rejected": -1.59375,
190
+ "step": 100
191
+ },
192
+ {
193
+ "epoch": 1.746031746031746,
194
+ "grad_norm": 16.88214421094986,
195
+ "learning_rate": 2.323529411764706e-07,
196
+ "logits/chosen": -2.3125,
197
+ "logits/rejected": -1.171875,
198
+ "logps/chosen": -55.75,
199
+ "logps/rejected": -90.5,
200
+ "loss": 0.2389,
201
+ "rewards/accuracies": 0.925000011920929,
202
+ "rewards/chosen": 0.197265625,
203
+ "rewards/margins": 1.9296875,
204
+ "rewards/rejected": -1.734375,
205
+ "step": 110
206
+ },
207
+ {
208
+ "epoch": 1.9047619047619047,
209
+ "grad_norm": 12.829485946331816,
210
+ "learning_rate": 2.0294117647058823e-07,
211
+ "logits/chosen": -1.8828125,
212
+ "logits/rejected": -0.9453125,
213
+ "logps/chosen": -50.5,
214
+ "logps/rejected": -102.5,
215
+ "loss": 0.2831,
216
+ "rewards/accuracies": 0.9125000238418579,
217
+ "rewards/chosen": 0.10888671875,
218
+ "rewards/margins": 2.578125,
219
+ "rewards/rejected": -2.46875,
220
+ "step": 120
221
+ },
222
+ {
223
+ "epoch": 2.0,
224
+ "eval_logits/chosen": -1.84375,
225
+ "eval_logits/rejected": -1.171875,
226
+ "eval_logps/chosen": -66.0,
227
+ "eval_logps/rejected": -97.5,
228
+ "eval_loss": 0.6284375190734863,
229
+ "eval_rewards/accuracies": 0.6071428656578064,
230
+ "eval_rewards/chosen": -0.453125,
231
+ "eval_rewards/margins": 0.75,
232
+ "eval_rewards/rejected": -1.203125,
233
+ "eval_runtime": 23.6296,
234
+ "eval_samples_per_second": 8.464,
235
+ "eval_steps_per_second": 0.296,
236
+ "step": 126
237
+ },
238
+ {
239
+ "epoch": 2.0634920634920633,
240
+ "grad_norm": 7.418629403738186,
241
+ "learning_rate": 1.7352941176470587e-07,
242
+ "logits/chosen": -2.109375,
243
+ "logits/rejected": -0.73046875,
244
+ "logps/chosen": -61.25,
245
+ "logps/rejected": -106.0,
246
+ "loss": 0.1749,
247
+ "rewards/accuracies": 0.9624999761581421,
248
+ "rewards/chosen": 0.4296875,
249
+ "rewards/margins": 2.625,
250
+ "rewards/rejected": -2.203125,
251
+ "step": 130
252
+ },
253
+ {
254
+ "epoch": 2.2222222222222223,
255
+ "grad_norm": 5.917946365968759,
256
+ "learning_rate": 1.441176470588235e-07,
257
+ "logits/chosen": -2.390625,
258
+ "logits/rejected": -1.2890625,
259
+ "logps/chosen": -72.0,
260
+ "logps/rejected": -100.0,
261
+ "loss": 0.1132,
262
+ "rewards/accuracies": 0.9750000238418579,
263
+ "rewards/chosen": 0.6015625,
264
+ "rewards/margins": 3.65625,
265
+ "rewards/rejected": -3.0625,
266
+ "step": 140
267
+ },
268
+ {
269
+ "epoch": 2.380952380952381,
270
+ "grad_norm": 7.62179839987552,
271
+ "learning_rate": 1.1470588235294116e-07,
272
+ "logits/chosen": -2.125,
273
+ "logits/rejected": -1.296875,
274
+ "logps/chosen": -54.0,
275
+ "logps/rejected": -92.5,
276
+ "loss": 0.125,
277
+ "rewards/accuracies": 0.987500011920929,
278
+ "rewards/chosen": 0.66796875,
279
+ "rewards/margins": 3.171875,
280
+ "rewards/rejected": -2.5,
281
+ "step": 150
282
+ },
283
+ {
284
+ "epoch": 2.5396825396825395,
285
+ "grad_norm": 7.689817277750616,
286
+ "learning_rate": 8.529411764705883e-08,
287
+ "logits/chosen": -1.8671875,
288
+ "logits/rejected": -2.3125,
289
+ "logps/chosen": -50.25,
290
+ "logps/rejected": -92.5,
291
+ "loss": 0.118,
292
+ "rewards/accuracies": 0.949999988079071,
293
+ "rewards/chosen": 0.3203125,
294
+ "rewards/margins": 3.109375,
295
+ "rewards/rejected": -2.78125,
296
+ "step": 160
297
+ },
298
+ {
299
+ "epoch": 2.6984126984126986,
300
+ "grad_norm": 5.583079792162674,
301
+ "learning_rate": 5.5882352941176474e-08,
302
+ "logits/chosen": -2.21875,
303
+ "logits/rejected": -1.1328125,
304
+ "logps/chosen": -66.0,
305
+ "logps/rejected": -114.5,
306
+ "loss": 0.145,
307
+ "rewards/accuracies": 0.9750000238418579,
308
+ "rewards/chosen": 0.388671875,
309
+ "rewards/margins": 3.671875,
310
+ "rewards/rejected": -3.28125,
311
+ "step": 170
312
+ },
313
+ {
314
+ "epoch": 2.857142857142857,
315
+ "grad_norm": 10.79399873370834,
316
+ "learning_rate": 2.6470588235294116e-08,
317
+ "logits/chosen": -1.6640625,
318
+ "logits/rejected": -1.5234375,
319
+ "logps/chosen": -59.0,
320
+ "logps/rejected": -87.5,
321
+ "loss": 0.1329,
322
+ "rewards/accuracies": 0.949999988079071,
323
+ "rewards/chosen": 0.310546875,
324
+ "rewards/margins": 2.75,
325
+ "rewards/rejected": -2.4375,
326
+ "step": 180
327
+ },
328
+ {
329
+ "epoch": 3.0,
330
+ "eval_logits/chosen": -2.21875,
331
+ "eval_logits/rejected": -1.4921875,
332
+ "eval_logps/chosen": -69.5,
333
+ "eval_logps/rejected": -102.5,
334
+ "eval_loss": 0.6340625286102295,
335
+ "eval_rewards/accuracies": 0.6607142686843872,
336
+ "eval_rewards/chosen": -0.76953125,
337
+ "eval_rewards/margins": 0.921875,
338
+ "eval_rewards/rejected": -1.6953125,
339
+ "eval_runtime": 21.8558,
340
+ "eval_samples_per_second": 9.151,
341
+ "eval_steps_per_second": 0.32,
342
+ "step": 189
343
+ },
344
+ {
345
+ "epoch": 3.0,
346
+ "step": 189,
347
+ "total_flos": 0.0,
348
+ "train_loss": 0.354867279214203,
349
+ "train_runtime": 2422.2159,
350
+ "train_samples_per_second": 2.477,
351
+ "train_steps_per_second": 0.078
352
+ }
353
+ ],
354
+ "logging_steps": 10,
355
+ "max_steps": 189,
356
+ "num_input_tokens_seen": 0,
357
+ "num_train_epochs": 3,
358
+ "save_steps": 500,
359
+ "stateful_callbacks": {
360
+ "TrainerControl": {
361
+ "args": {
362
+ "should_epoch_stop": false,
363
+ "should_evaluate": false,
364
+ "should_log": false,
365
+ "should_save": true,
366
+ "should_training_stop": true
367
+ },
368
+ "attributes": {}
369
+ }
370
+ },
371
+ "total_flos": 0.0,
372
+ "train_batch_size": 8,
373
+ "trial_name": null,
374
+ "trial_params": null
375
+ }