Hugging Face's logo

AmberYifan
/
Llama-2-7b-relevant-dpo

Text Generation
Transformers
Safetensors
llama
Generated from Trainer
trl
dpo
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
AmberYifan commited on
Commit
d87f2ac
·
verified ·
1 Parent(s): df9028a

Model save

Browse files
Files changed (5) hide show
  1. README.md +68 -0
  2. all_results.json +9 -0
  3. generation_config.json +10 -0
  4. train_results.json +9 -0
  5. trainer_state.json +375 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: AmberYifan/llama2-7b-sft-ultrachat-safeRLHF
3
+ library_name: transformers
4
+ model_name: Llama-2-7b-relevant-dpo
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - dpo
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for Llama-2-7b-relevant-dpo
13
+
14
+ This model is a fine-tuned version of [AmberYifan/llama2-7b-sft-ultrachat-safeRLHF](https://huggingface.co/AmberYifan/llama2-7b-sft-ultrachat-safeRLHF).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="AmberYifan/Llama-2-7b-relevant-dpo", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/yifanwang/huggingface/runs/ff28obuu)
31
+
32
+ This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
33
+
34
+ ### Framework versions
35
+
36
+ - TRL: 0.12.2
37
+ - Transformers: 4.46.3
38
+ - Pytorch: 2.5.1+cu118
39
+ - Datasets: 3.2.0
40
+ - Tokenizers: 0.20.3
41
+
42
+ ## Citations
43
+
44
+ Cite DPO as:
45
+
46
+ ```bibtex
47
+ @inproceedings{rafailov2023direct,
48
+ title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
49
+ author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
50
+ year = 2023,
51
+ booktitle = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
52
+ url = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
53
+ editor = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
54
+ }
55
+ ```
56
+
57
+ Cite TRL as:
58
+
59
+ ```bibtex
60
+ @misc{vonwerra2022trl,
61
+ title = {{TRL: Transformer Reinforcement Learning}},
62
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
63
+ year = 2020,
64
+ journal = {GitHub repository},
65
+ publisher = {GitHub},
66
+ howpublished = {\url{https://github.com/huggingface/trl}}
67
+ }
68
+ ```
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.48406885540674605,
5
+ "train_runtime": 1618.971,
6
+ "train_samples": 2000,
7
+ "train_samples_per_second": 3.706,
8
+ "train_steps_per_second": 0.117
9
+ }
generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 1,
3
+ "do_sample": true,
4
+ "eos_token_id": 2,
5
+ "max_length": 4096,
6
+ "pad_token_id": 0,
7
+ "temperature": 0.6,
8
+ "top_p": 0.9,
9
+ "transformers_version": "4.46.3"
10
+ }
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.48406885540674605,
5
+ "train_runtime": 1618.971,
6
+ "train_samples": 2000,
7
+ "train_samples_per_second": 3.706,
8
+ "train_steps_per_second": 0.117
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,375 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 3.0,
5
+ "eval_steps": 500,
6
+ "global_step": 189,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.015873015873015872,
13
+ "grad_norm": 16.2200460330357,
14
+ "learning_rate": 2.6315789473684208e-08,
15
+ "logits/chosen": -1.5234375,
16
+ "logits/rejected": -1.4375,
17
+ "logps/chosen": -166.0,
18
+ "logps/rejected": -129.0,
19
+ "loss": 0.6914,
20
+ "rewards/accuracies": 0.0,
21
+ "rewards/chosen": 0.0,
22
+ "rewards/margins": 0.0,
23
+ "rewards/rejected": 0.0,
24
+ "step": 1
25
+ },
26
+ {
27
+ "epoch": 0.15873015873015872,
28
+ "grad_norm": 16.23804389503029,
29
+ "learning_rate": 2.631578947368421e-07,
30
+ "logits/chosen": -1.53125,
31
+ "logits/rejected": -1.53125,
32
+ "logps/chosen": -144.0,
33
+ "logps/rejected": -120.0,
34
+ "loss": 0.6946,
35
+ "rewards/accuracies": 0.2361111044883728,
36
+ "rewards/chosen": -0.000339508056640625,
37
+ "rewards/margins": -0.000701904296875,
38
+ "rewards/rejected": 0.0003509521484375,
39
+ "step": 10
40
+ },
41
+ {
42
+ "epoch": 0.31746031746031744,
43
+ "grad_norm": 15.80676425202867,
44
+ "learning_rate": 4.970588235294118e-07,
45
+ "logits/chosen": -1.609375,
46
+ "logits/rejected": -1.5234375,
47
+ "logps/chosen": -162.0,
48
+ "logps/rejected": -133.0,
49
+ "loss": 0.6844,
50
+ "rewards/accuracies": 0.4625000059604645,
51
+ "rewards/chosen": -0.026611328125,
52
+ "rewards/margins": 0.0341796875,
53
+ "rewards/rejected": -0.060791015625,
54
+ "step": 20
55
+ },
56
+ {
57
+ "epoch": 0.47619047619047616,
58
+ "grad_norm": 13.971741456622834,
59
+ "learning_rate": 4.676470588235294e-07,
60
+ "logits/chosen": -1.59375,
61
+ "logits/rejected": -1.546875,
62
+ "logps/chosen": -151.0,
63
+ "logps/rejected": -133.0,
64
+ "loss": 0.6601,
65
+ "rewards/accuracies": 0.5874999761581421,
66
+ "rewards/chosen": -0.10693359375,
67
+ "rewards/margins": 0.07177734375,
68
+ "rewards/rejected": -0.1787109375,
69
+ "step": 30
70
+ },
71
+ {
72
+ "epoch": 0.6349206349206349,
73
+ "grad_norm": 14.439143279421419,
74
+ "learning_rate": 4.38235294117647e-07,
75
+ "logits/chosen": -1.625,
76
+ "logits/rejected": -1.578125,
77
+ "logps/chosen": -165.0,
78
+ "logps/rejected": -145.0,
79
+ "loss": 0.6307,
80
+ "rewards/accuracies": 0.6875,
81
+ "rewards/chosen": -0.158203125,
82
+ "rewards/margins": 0.14453125,
83
+ "rewards/rejected": -0.302734375,
84
+ "step": 40
85
+ },
86
+ {
87
+ "epoch": 0.7936507936507936,
88
+ "grad_norm": 12.933116251658811,
89
+ "learning_rate": 4.0882352941176465e-07,
90
+ "logits/chosen": -1.5703125,
91
+ "logits/rejected": -1.5078125,
92
+ "logps/chosen": -155.0,
93
+ "logps/rejected": -134.0,
94
+ "loss": 0.6348,
95
+ "rewards/accuracies": 0.7124999761581421,
96
+ "rewards/chosen": -0.298828125,
97
+ "rewards/margins": 0.203125,
98
+ "rewards/rejected": -0.50390625,
99
+ "step": 50
100
+ },
101
+ {
102
+ "epoch": 0.9523809523809523,
103
+ "grad_norm": 13.17457815976231,
104
+ "learning_rate": 3.7941176470588235e-07,
105
+ "logits/chosen": -1.5390625,
106
+ "logits/rejected": -1.53125,
107
+ "logps/chosen": -157.0,
108
+ "logps/rejected": -148.0,
109
+ "loss": 0.607,
110
+ "rewards/accuracies": 0.675000011920929,
111
+ "rewards/chosen": -0.390625,
112
+ "rewards/margins": 0.1884765625,
113
+ "rewards/rejected": -0.578125,
114
+ "step": 60
115
+ },
116
+ {
117
+ "epoch": 1.0,
118
+ "eval_logits/chosen": -1.609375,
119
+ "eval_logits/rejected": -1.453125,
120
+ "eval_logps/chosen": -161.0,
121
+ "eval_logps/rejected": -117.5,
122
+ "eval_loss": 0.6323873400688171,
123
+ "eval_rewards/accuracies": 0.7250000238418579,
124
+ "eval_rewards/chosen": -0.5625,
125
+ "eval_rewards/margins": 0.2578125,
126
+ "eval_rewards/rejected": -0.8203125,
127
+ "eval_runtime": 8.7111,
128
+ "eval_samples_per_second": 16.875,
129
+ "eval_steps_per_second": 0.574,
130
+ "step": 63
131
+ },
132
+ {
133
+ "epoch": 1.1111111111111112,
134
+ "grad_norm": 10.940163753846868,
135
+ "learning_rate": 3.5e-07,
136
+ "logits/chosen": -1.5625,
137
+ "logits/rejected": -1.5234375,
138
+ "logps/chosen": -151.0,
139
+ "logps/rejected": -126.5,
140
+ "loss": 0.5062,
141
+ "rewards/accuracies": 0.8999999761581421,
142
+ "rewards/chosen": -0.4453125,
143
+ "rewards/margins": 0.46875,
144
+ "rewards/rejected": -0.9140625,
145
+ "step": 70
146
+ },
147
+ {
148
+ "epoch": 1.2698412698412698,
149
+ "grad_norm": 10.448217147222541,
150
+ "learning_rate": 3.205882352941177e-07,
151
+ "logits/chosen": -1.515625,
152
+ "logits/rejected": -1.4609375,
153
+ "logps/chosen": -142.0,
154
+ "logps/rejected": -123.5,
155
+ "loss": 0.4673,
156
+ "rewards/accuracies": 0.9125000238418579,
157
+ "rewards/chosen": -0.5625,
158
+ "rewards/margins": 0.57421875,
159
+ "rewards/rejected": -1.1328125,
160
+ "step": 80
161
+ },
162
+ {
163
+ "epoch": 1.4285714285714286,
164
+ "grad_norm": 10.288072810717312,
165
+ "learning_rate": 2.911764705882353e-07,
166
+ "logits/chosen": -1.578125,
167
+ "logits/rejected": -1.53125,
168
+ "logps/chosen": -157.0,
169
+ "logps/rejected": -147.0,
170
+ "loss": 0.4288,
171
+ "rewards/accuracies": 0.887499988079071,
172
+ "rewards/chosen": -0.53515625,
173
+ "rewards/margins": 0.6953125,
174
+ "rewards/rejected": -1.2265625,
175
+ "step": 90
176
+ },
177
+ {
178
+ "epoch": 1.5873015873015874,
179
+ "grad_norm": 10.019896435666249,
180
+ "learning_rate": 2.6176470588235295e-07,
181
+ "logits/chosen": -1.609375,
182
+ "logits/rejected": -1.6015625,
183
+ "logps/chosen": -162.0,
184
+ "logps/rejected": -155.0,
185
+ "loss": 0.4419,
186
+ "rewards/accuracies": 0.862500011920929,
187
+ "rewards/chosen": -0.640625,
188
+ "rewards/margins": 0.65625,
189
+ "rewards/rejected": -1.296875,
190
+ "step": 100
191
+ },
192
+ {
193
+ "epoch": 1.746031746031746,
194
+ "grad_norm": 12.080281395370564,
195
+ "learning_rate": 2.323529411764706e-07,
196
+ "logits/chosen": -1.5625,
197
+ "logits/rejected": -1.546875,
198
+ "logps/chosen": -156.0,
199
+ "logps/rejected": -148.0,
200
+ "loss": 0.4641,
201
+ "rewards/accuracies": 0.862500011920929,
202
+ "rewards/chosen": -0.53515625,
203
+ "rewards/margins": 0.79296875,
204
+ "rewards/rejected": -1.328125,
205
+ "step": 110
206
+ },
207
+ {
208
+ "epoch": 1.9047619047619047,
209
+ "grad_norm": 11.329269122728626,
210
+ "learning_rate": 2.0294117647058823e-07,
211
+ "logits/chosen": -1.4921875,
212
+ "logits/rejected": -1.5078125,
213
+ "logps/chosen": -140.0,
214
+ "logps/rejected": -145.0,
215
+ "loss": 0.4385,
216
+ "rewards/accuracies": 0.7875000238418579,
217
+ "rewards/chosen": -0.5390625,
218
+ "rewards/margins": 0.63671875,
219
+ "rewards/rejected": -1.171875,
220
+ "step": 120
221
+ },
222
+ {
223
+ "epoch": 2.0,
224
+ "eval_logits/chosen": -1.6015625,
225
+ "eval_logits/rejected": -1.453125,
226
+ "eval_logps/chosen": -161.0,
227
+ "eval_logps/rejected": -119.5,
228
+ "eval_loss": 0.6192867755889893,
229
+ "eval_rewards/accuracies": 0.7250000238418579,
230
+ "eval_rewards/chosen": -0.625,
231
+ "eval_rewards/margins": 0.3515625,
232
+ "eval_rewards/rejected": -0.9765625,
233
+ "eval_runtime": 10.8318,
234
+ "eval_samples_per_second": 13.571,
235
+ "eval_steps_per_second": 0.462,
236
+ "step": 126
237
+ },
238
+ {
239
+ "epoch": 2.0634920634920633,
240
+ "grad_norm": 9.801925381761396,
241
+ "learning_rate": 1.7352941176470587e-07,
242
+ "logits/chosen": -1.5625,
243
+ "logits/rejected": -1.4921875,
244
+ "logps/chosen": -159.0,
245
+ "logps/rejected": -144.0,
246
+ "loss": 0.404,
247
+ "rewards/accuracies": 0.887499988079071,
248
+ "rewards/chosen": -0.263671875,
249
+ "rewards/margins": 0.90234375,
250
+ "rewards/rejected": -1.1640625,
251
+ "step": 130
252
+ },
253
+ {
254
+ "epoch": 2.2222222222222223,
255
+ "grad_norm": 10.394699460580199,
256
+ "learning_rate": 1.441176470588235e-07,
257
+ "logits/chosen": -1.5859375,
258
+ "logits/rejected": -1.53125,
259
+ "logps/chosen": -152.0,
260
+ "logps/rejected": -146.0,
261
+ "loss": 0.3731,
262
+ "rewards/accuracies": 0.9750000238418579,
263
+ "rewards/chosen": -0.271484375,
264
+ "rewards/margins": 0.96875,
265
+ "rewards/rejected": -1.2421875,
266
+ "step": 140
267
+ },
268
+ {
269
+ "epoch": 2.380952380952381,
270
+ "grad_norm": 7.8219119833188975,
271
+ "learning_rate": 1.1470588235294116e-07,
272
+ "logits/chosen": -1.5390625,
273
+ "logits/rejected": -1.453125,
274
+ "logps/chosen": -162.0,
275
+ "logps/rejected": -132.0,
276
+ "loss": 0.3477,
277
+ "rewards/accuracies": 0.9125000238418579,
278
+ "rewards/chosen": -0.376953125,
279
+ "rewards/margins": 0.90625,
280
+ "rewards/rejected": -1.28125,
281
+ "step": 150
282
+ },
283
+ {
284
+ "epoch": 2.5396825396825395,
285
+ "grad_norm": 8.679287106154373,
286
+ "learning_rate": 8.529411764705883e-08,
287
+ "logits/chosen": -1.5625,
288
+ "logits/rejected": -1.4921875,
289
+ "logps/chosen": -163.0,
290
+ "logps/rejected": -153.0,
291
+ "loss": 0.3479,
292
+ "rewards/accuracies": 0.949999988079071,
293
+ "rewards/chosen": -0.3046875,
294
+ "rewards/margins": 1.125,
295
+ "rewards/rejected": -1.4296875,
296
+ "step": 160
297
+ },
298
+ {
299
+ "epoch": 2.6984126984126986,
300
+ "grad_norm": 8.351083925741897,
301
+ "learning_rate": 5.5882352941176474e-08,
302
+ "logits/chosen": -1.5078125,
303
+ "logits/rejected": -1.53125,
304
+ "logps/chosen": -146.0,
305
+ "logps/rejected": -157.0,
306
+ "loss": 0.3607,
307
+ "rewards/accuracies": 0.925000011920929,
308
+ "rewards/chosen": -0.4609375,
309
+ "rewards/margins": 0.9375,
310
+ "rewards/rejected": -1.3984375,
311
+ "step": 170
312
+ },
313
+ {
314
+ "epoch": 2.857142857142857,
315
+ "grad_norm": 7.683812751753157,
316
+ "learning_rate": 2.6470588235294116e-08,
317
+ "logits/chosen": -1.53125,
318
+ "logits/rejected": -1.546875,
319
+ "logps/chosen": -154.0,
320
+ "logps/rejected": -157.0,
321
+ "loss": 0.3708,
322
+ "rewards/accuracies": 0.9125000238418579,
323
+ "rewards/chosen": -0.408203125,
324
+ "rewards/margins": 0.96875,
325
+ "rewards/rejected": -1.375,
326
+ "step": 180
327
+ },
328
+ {
329
+ "epoch": 3.0,
330
+ "eval_logits/chosen": -1.6015625,
331
+ "eval_logits/rejected": -1.453125,
332
+ "eval_logps/chosen": -161.0,
333
+ "eval_logps/rejected": -120.5,
334
+ "eval_loss": 0.6126700639724731,
335
+ "eval_rewards/accuracies": 0.75,
336
+ "eval_rewards/chosen": -0.67578125,
337
+ "eval_rewards/margins": 0.42578125,
338
+ "eval_rewards/rejected": -1.1015625,
339
+ "eval_runtime": 13.2152,
340
+ "eval_samples_per_second": 11.124,
341
+ "eval_steps_per_second": 0.378,
342
+ "step": 189
343
+ },
344
+ {
345
+ "epoch": 3.0,
346
+ "step": 189,
347
+ "total_flos": 0.0,
348
+ "train_loss": 0.48406885540674605,
349
+ "train_runtime": 1618.971,
350
+ "train_samples_per_second": 3.706,
351
+ "train_steps_per_second": 0.117
352
+ }
353
+ ],
354
+ "logging_steps": 10,
355
+ "max_steps": 189,
356
+ "num_input_tokens_seen": 0,
357
+ "num_train_epochs": 3,
358
+ "save_steps": 500,
359
+ "stateful_callbacks": {
360
+ "TrainerControl": {
361
+ "args": {
362
+ "should_epoch_stop": false,
363
+ "should_evaluate": false,
364
+ "should_log": false,
365
+ "should_save": true,
366
+ "should_training_stop": true
367
+ },
368
+ "attributes": {}
369
+ }
370
+ },
371
+ "total_flos": 0.0,
372
+ "train_batch_size": 8,
373
+ "trial_name": null,
374
+ "trial_params": null
375
+ }