Hugging Face's logo

AmberYifan
/
Llama-2-7b-sft-DPO-2k

Text Generation
Transformers
Safetensors
llama
Generated from Trainer
trl
dpo
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
AmberYifan commited on
Commit
47cc370
·
verified ·
1 Parent(s): f9dcd3f

Model save

Browse files
Files changed (5) hide show
  1. README.md +68 -0
  2. all_results.json +9 -0
  3. generation_config.json +10 -0
  4. train_results.json +9 -0
  5. trainer_state.json +375 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: AmberYifan/llama2-7b-sft-ultrachat-safeRLHF
3
+ library_name: transformers
4
+ model_name: Llama-2-7b-sft-DPO-2k
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - dpo
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for Llama-2-7b-sft-DPO-2k
13
+
14
+ This model is a fine-tuned version of [AmberYifan/llama2-7b-sft-ultrachat-safeRLHF](https://huggingface.co/AmberYifan/llama2-7b-sft-ultrachat-safeRLHF).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="AmberYifan/Llama-2-7b-sft-DPO-2k", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/yifanwang/huggingface/runs/l5hkbfwb)
31
+
32
+ This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
33
+
34
+ ### Framework versions
35
+
36
+ - TRL: 0.12.2
37
+ - Transformers: 4.46.3
38
+ - Pytorch: 2.5.1+cu118
39
+ - Datasets: 3.2.0
40
+ - Tokenizers: 0.20.3
41
+
42
+ ## Citations
43
+
44
+ Cite DPO as:
45
+
46
+ ```bibtex
47
+ @inproceedings{rafailov2023direct,
48
+ title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
49
+ author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
50
+ year = 2023,
51
+ booktitle = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
52
+ url = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
53
+ editor = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
54
+ }
55
+ ```
56
+
57
+ Cite TRL as:
58
+
59
+ ```bibtex
60
+ @misc{vonwerra2022trl,
61
+ title = {{TRL: Transformer Reinforcement Learning}},
62
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
63
+ year = 2020,
64
+ journal = {GitHub repository},
65
+ publisher = {GitHub},
66
+ howpublished = {\url{https://github.com/huggingface/trl}}
67
+ }
68
+ ```
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.5609137318121693,
5
+ "train_runtime": 1965.8465,
6
+ "train_samples": 1999,
7
+ "train_samples_per_second": 3.051,
8
+ "train_steps_per_second": 0.096
9
+ }
generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 1,
3
+ "do_sample": true,
4
+ "eos_token_id": 2,
5
+ "max_length": 4096,
6
+ "pad_token_id": 0,
7
+ "temperature": 0.6,
8
+ "top_p": 0.9,
9
+ "transformers_version": "4.46.3"
10
+ }
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.5609137318121693,
5
+ "train_runtime": 1965.8465,
6
+ "train_samples": 1999,
7
+ "train_samples_per_second": 3.051,
8
+ "train_steps_per_second": 0.096
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,375 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 3.0,
5
+ "eval_steps": 500,
6
+ "global_step": 189,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.015873015873015872,
13
+ "grad_norm": 15.294358962776673,
14
+ "learning_rate": 2.6315789473684208e-08,
15
+ "logits/chosen": -1.6171875,
16
+ "logits/rejected": -1.4296875,
17
+ "logps/chosen": -139.0,
18
+ "logps/rejected": -104.5,
19
+ "loss": 0.6914,
20
+ "rewards/accuracies": 0.0,
21
+ "rewards/chosen": 0.0,
22
+ "rewards/margins": 0.0,
23
+ "rewards/rejected": 0.0,
24
+ "step": 1
25
+ },
26
+ {
27
+ "epoch": 0.15873015873015872,
28
+ "grad_norm": 15.094161248794443,
29
+ "learning_rate": 2.631578947368421e-07,
30
+ "logits/chosen": -1.5859375,
31
+ "logits/rejected": -1.53125,
32
+ "logps/chosen": -149.0,
33
+ "logps/rejected": -129.0,
34
+ "loss": 0.6923,
35
+ "rewards/accuracies": 0.2222222238779068,
36
+ "rewards/chosen": -0.0013885498046875,
37
+ "rewards/margins": 0.0045166015625,
38
+ "rewards/rejected": -0.005889892578125,
39
+ "step": 10
40
+ },
41
+ {
42
+ "epoch": 0.31746031746031744,
43
+ "grad_norm": 15.606155891832502,
44
+ "learning_rate": 4.970588235294118e-07,
45
+ "logits/chosen": -1.5703125,
46
+ "logits/rejected": -1.546875,
47
+ "logps/chosen": -152.0,
48
+ "logps/rejected": -133.0,
49
+ "loss": 0.6871,
50
+ "rewards/accuracies": 0.3125,
51
+ "rewards/chosen": -0.031005859375,
52
+ "rewards/margins": 0.00125885009765625,
53
+ "rewards/rejected": -0.0322265625,
54
+ "step": 20
55
+ },
56
+ {
57
+ "epoch": 0.47619047619047616,
58
+ "grad_norm": 14.846625318284318,
59
+ "learning_rate": 4.676470588235294e-07,
60
+ "logits/chosen": -1.6015625,
61
+ "logits/rejected": -1.609375,
62
+ "logps/chosen": -159.0,
63
+ "logps/rejected": -149.0,
64
+ "loss": 0.6824,
65
+ "rewards/accuracies": 0.4625000059604645,
66
+ "rewards/chosen": -0.06298828125,
67
+ "rewards/margins": 0.0184326171875,
68
+ "rewards/rejected": -0.08154296875,
69
+ "step": 30
70
+ },
71
+ {
72
+ "epoch": 0.6349206349206349,
73
+ "grad_norm": 12.673127513530932,
74
+ "learning_rate": 4.38235294117647e-07,
75
+ "logits/chosen": -1.59375,
76
+ "logits/rejected": -1.578125,
77
+ "logps/chosen": -170.0,
78
+ "logps/rejected": -155.0,
79
+ "loss": 0.6793,
80
+ "rewards/accuracies": 0.5,
81
+ "rewards/chosen": -0.0751953125,
82
+ "rewards/margins": 0.042236328125,
83
+ "rewards/rejected": -0.11767578125,
84
+ "step": 40
85
+ },
86
+ {
87
+ "epoch": 0.7936507936507936,
88
+ "grad_norm": 14.318265772228457,
89
+ "learning_rate": 4.0882352941176465e-07,
90
+ "logits/chosen": -1.59375,
91
+ "logits/rejected": -1.546875,
92
+ "logps/chosen": -146.0,
93
+ "logps/rejected": -126.5,
94
+ "loss": 0.6741,
95
+ "rewards/accuracies": 0.612500011920929,
96
+ "rewards/chosen": -0.1357421875,
97
+ "rewards/margins": 0.0634765625,
98
+ "rewards/rejected": -0.19921875,
99
+ "step": 50
100
+ },
101
+ {
102
+ "epoch": 0.9523809523809523,
103
+ "grad_norm": 14.53000414550404,
104
+ "learning_rate": 3.7941176470588235e-07,
105
+ "logits/chosen": -1.5234375,
106
+ "logits/rejected": -1.5,
107
+ "logps/chosen": -138.0,
108
+ "logps/rejected": -138.0,
109
+ "loss": 0.6696,
110
+ "rewards/accuracies": 0.4000000059604645,
111
+ "rewards/chosen": -0.1845703125,
112
+ "rewards/margins": 0.024169921875,
113
+ "rewards/rejected": -0.208984375,
114
+ "step": 60
115
+ },
116
+ {
117
+ "epoch": 1.0,
118
+ "eval_logits/chosen": -1.5,
119
+ "eval_logits/rejected": -1.4921875,
120
+ "eval_logps/chosen": -140.0,
121
+ "eval_logps/rejected": -127.0,
122
+ "eval_loss": 0.6568750143051147,
123
+ "eval_rewards/accuracies": 0.5714285969734192,
124
+ "eval_rewards/chosen": -0.1796875,
125
+ "eval_rewards/margins": 0.0966796875,
126
+ "eval_rewards/rejected": -0.27734375,
127
+ "eval_runtime": 14.7253,
128
+ "eval_samples_per_second": 13.582,
129
+ "eval_steps_per_second": 0.475,
130
+ "step": 63
131
+ },
132
+ {
133
+ "epoch": 1.1111111111111112,
134
+ "grad_norm": 13.811509421726951,
135
+ "learning_rate": 3.5e-07,
136
+ "logits/chosen": -1.53125,
137
+ "logits/rejected": -1.546875,
138
+ "logps/chosen": -142.0,
139
+ "logps/rejected": -147.0,
140
+ "loss": 0.5957,
141
+ "rewards/accuracies": 0.675000011920929,
142
+ "rewards/chosen": -0.1337890625,
143
+ "rewards/margins": 0.2177734375,
144
+ "rewards/rejected": -0.3515625,
145
+ "step": 70
146
+ },
147
+ {
148
+ "epoch": 1.2698412698412698,
149
+ "grad_norm": 11.993229269242237,
150
+ "learning_rate": 3.205882352941177e-07,
151
+ "logits/chosen": -1.5546875,
152
+ "logits/rejected": -1.5390625,
153
+ "logps/chosen": -150.0,
154
+ "logps/rejected": -135.0,
155
+ "loss": 0.5614,
156
+ "rewards/accuracies": 0.800000011920929,
157
+ "rewards/chosen": -0.1328125,
158
+ "rewards/margins": 0.28515625,
159
+ "rewards/rejected": -0.41796875,
160
+ "step": 80
161
+ },
162
+ {
163
+ "epoch": 1.4285714285714286,
164
+ "grad_norm": 11.581245327928885,
165
+ "learning_rate": 2.911764705882353e-07,
166
+ "logits/chosen": -1.5703125,
167
+ "logits/rejected": -1.5078125,
168
+ "logps/chosen": -144.0,
169
+ "logps/rejected": -133.0,
170
+ "loss": 0.5406,
171
+ "rewards/accuracies": 0.8125,
172
+ "rewards/chosen": -0.2333984375,
173
+ "rewards/margins": 0.341796875,
174
+ "rewards/rejected": -0.57421875,
175
+ "step": 90
176
+ },
177
+ {
178
+ "epoch": 1.5873015873015874,
179
+ "grad_norm": 12.825913209383442,
180
+ "learning_rate": 2.6176470588235295e-07,
181
+ "logits/chosen": -1.5703125,
182
+ "logits/rejected": -1.53125,
183
+ "logps/chosen": -152.0,
184
+ "logps/rejected": -138.0,
185
+ "loss": 0.5382,
186
+ "rewards/accuracies": 0.762499988079071,
187
+ "rewards/chosen": -0.26953125,
188
+ "rewards/margins": 0.390625,
189
+ "rewards/rejected": -0.66015625,
190
+ "step": 100
191
+ },
192
+ {
193
+ "epoch": 1.746031746031746,
194
+ "grad_norm": 12.686457389394935,
195
+ "learning_rate": 2.323529411764706e-07,
196
+ "logits/chosen": -1.5703125,
197
+ "logits/rejected": -1.5625,
198
+ "logps/chosen": -160.0,
199
+ "logps/rejected": -154.0,
200
+ "loss": 0.5295,
201
+ "rewards/accuracies": 0.9125000238418579,
202
+ "rewards/chosen": -0.23046875,
203
+ "rewards/margins": 0.46484375,
204
+ "rewards/rejected": -0.6953125,
205
+ "step": 110
206
+ },
207
+ {
208
+ "epoch": 1.9047619047619047,
209
+ "grad_norm": 11.815278327307368,
210
+ "learning_rate": 2.0294117647058823e-07,
211
+ "logits/chosen": -1.5546875,
212
+ "logits/rejected": -1.5,
213
+ "logps/chosen": -138.0,
214
+ "logps/rejected": -137.0,
215
+ "loss": 0.5284,
216
+ "rewards/accuracies": 0.800000011920929,
217
+ "rewards/chosen": -0.33984375,
218
+ "rewards/margins": 0.365234375,
219
+ "rewards/rejected": -0.70703125,
220
+ "step": 120
221
+ },
222
+ {
223
+ "epoch": 2.0,
224
+ "eval_logits/chosen": -1.484375,
225
+ "eval_logits/rejected": -1.4765625,
226
+ "eval_logps/chosen": -143.0,
227
+ "eval_logps/rejected": -130.0,
228
+ "eval_loss": 0.6310937404632568,
229
+ "eval_rewards/accuracies": 0.625,
230
+ "eval_rewards/chosen": -0.388671875,
231
+ "eval_rewards/margins": 0.185546875,
232
+ "eval_rewards/rejected": -0.57421875,
233
+ "eval_runtime": 17.8485,
234
+ "eval_samples_per_second": 11.205,
235
+ "eval_steps_per_second": 0.392,
236
+ "step": 126
237
+ },
238
+ {
239
+ "epoch": 2.0634920634920633,
240
+ "grad_norm": 10.190155022883303,
241
+ "learning_rate": 1.7352941176470587e-07,
242
+ "logits/chosen": -1.5703125,
243
+ "logits/rejected": -1.515625,
244
+ "logps/chosen": -154.0,
245
+ "logps/rejected": -146.0,
246
+ "loss": 0.4902,
247
+ "rewards/accuracies": 0.875,
248
+ "rewards/chosen": -0.298828125,
249
+ "rewards/margins": 0.51953125,
250
+ "rewards/rejected": -0.8203125,
251
+ "step": 130
252
+ },
253
+ {
254
+ "epoch": 2.2222222222222223,
255
+ "grad_norm": 11.482317016645608,
256
+ "learning_rate": 1.441176470588235e-07,
257
+ "logits/chosen": -1.5703125,
258
+ "logits/rejected": -1.5390625,
259
+ "logps/chosen": -146.0,
260
+ "logps/rejected": -147.0,
261
+ "loss": 0.4618,
262
+ "rewards/accuracies": 0.862500011920929,
263
+ "rewards/chosen": -0.298828125,
264
+ "rewards/margins": 0.57421875,
265
+ "rewards/rejected": -0.875,
266
+ "step": 140
267
+ },
268
+ {
269
+ "epoch": 2.380952380952381,
270
+ "grad_norm": 10.664901408549657,
271
+ "learning_rate": 1.1470588235294116e-07,
272
+ "logits/chosen": -1.484375,
273
+ "logits/rejected": -1.484375,
274
+ "logps/chosen": -139.0,
275
+ "logps/rejected": -135.0,
276
+ "loss": 0.4529,
277
+ "rewards/accuracies": 0.8999999761581421,
278
+ "rewards/chosen": -0.392578125,
279
+ "rewards/margins": 0.56640625,
280
+ "rewards/rejected": -0.95703125,
281
+ "step": 150
282
+ },
283
+ {
284
+ "epoch": 2.5396825396825395,
285
+ "grad_norm": 11.135694541597557,
286
+ "learning_rate": 8.529411764705883e-08,
287
+ "logits/chosen": -1.546875,
288
+ "logits/rejected": -1.46875,
289
+ "logps/chosen": -138.0,
290
+ "logps/rejected": -125.5,
291
+ "loss": 0.4585,
292
+ "rewards/accuracies": 0.8999999761581421,
293
+ "rewards/chosen": -0.34375,
294
+ "rewards/margins": 0.62109375,
295
+ "rewards/rejected": -0.96484375,
296
+ "step": 160
297
+ },
298
+ {
299
+ "epoch": 2.6984126984126986,
300
+ "grad_norm": 9.969592838659052,
301
+ "learning_rate": 5.5882352941176474e-08,
302
+ "logits/chosen": -1.515625,
303
+ "logits/rejected": -1.515625,
304
+ "logps/chosen": -144.0,
305
+ "logps/rejected": -147.0,
306
+ "loss": 0.4698,
307
+ "rewards/accuracies": 0.8999999761581421,
308
+ "rewards/chosen": -0.359375,
309
+ "rewards/margins": 0.60546875,
310
+ "rewards/rejected": -0.96484375,
311
+ "step": 170
312
+ },
313
+ {
314
+ "epoch": 2.857142857142857,
315
+ "grad_norm": 10.412087107819227,
316
+ "learning_rate": 2.6470588235294116e-08,
317
+ "logits/chosen": -1.5234375,
318
+ "logits/rejected": -1.4921875,
319
+ "logps/chosen": -147.0,
320
+ "logps/rejected": -137.0,
321
+ "loss": 0.4529,
322
+ "rewards/accuracies": 0.9125000238418579,
323
+ "rewards/chosen": -0.267578125,
324
+ "rewards/margins": 0.63671875,
325
+ "rewards/rejected": -0.90234375,
326
+ "step": 180
327
+ },
328
+ {
329
+ "epoch": 3.0,
330
+ "eval_logits/chosen": -1.484375,
331
+ "eval_logits/rejected": -1.46875,
332
+ "eval_logps/chosen": -143.0,
333
+ "eval_logps/rejected": -131.0,
334
+ "eval_loss": 0.6217187643051147,
335
+ "eval_rewards/accuracies": 0.6785714030265808,
336
+ "eval_rewards/chosen": -0.462890625,
337
+ "eval_rewards/margins": 0.220703125,
338
+ "eval_rewards/rejected": -0.68359375,
339
+ "eval_runtime": 17.9909,
340
+ "eval_samples_per_second": 11.117,
341
+ "eval_steps_per_second": 0.389,
342
+ "step": 189
343
+ },
344
+ {
345
+ "epoch": 3.0,
346
+ "step": 189,
347
+ "total_flos": 0.0,
348
+ "train_loss": 0.5609137318121693,
349
+ "train_runtime": 1965.8465,
350
+ "train_samples_per_second": 3.051,
351
+ "train_steps_per_second": 0.096
352
+ }
353
+ ],
354
+ "logging_steps": 10,
355
+ "max_steps": 189,
356
+ "num_input_tokens_seen": 0,
357
+ "num_train_epochs": 3,
358
+ "save_steps": 500,
359
+ "stateful_callbacks": {
360
+ "TrainerControl": {
361
+ "args": {
362
+ "should_epoch_stop": false,
363
+ "should_evaluate": false,
364
+ "should_log": false,
365
+ "should_save": true,
366
+ "should_training_stop": true
367
+ },
368
+ "attributes": {}
369
+ }
370
+ },
371
+ "total_flos": 0.0,
372
+ "train_batch_size": 8,
373
+ "trial_name": null,
374
+ "trial_params": null
375
+ }