English
yintongl commited on
Commit
5eb5bd6
·
verified ·
1 Parent(s): 4d64bd1

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -15
README.md CHANGED
@@ -35,25 +35,24 @@ print(tokenizer.decode(model.generate(**tokenizer("There is a girl who likes adv
35
  Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d
36
 
37
  ```bash
38
- lm_eval --model hf --model_args pretrained="Intel/gpt-j-6b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,mmlu --batch_size 32
39
  ```
40
 
41
 
42
-
43
- | Metric | BF16 | INT4 |
44
  | -------------- | ------ | ------ |
45
- | Avg. | 0. | 0.4993 |
46
- | mmlu | 0. | 0.2750 |
47
- | lambada_openai | 0. | 0.6845 |
48
- | hellaswag | 0. | 0.4897 |
49
- | winogrande | 0. | 0.6425 |
50
- | piqa | 0. | 0.7448 |
51
- | truthfulqa_mc1 | 0. | 0.2093 |
52
- | openbookqa | 0. | 0.2820 |
53
- | boolq | 0. | 0.6511 |
54
- | arc_easy | 0. | 0.6709 |
55
- | arc_challenge | 0. | 0.3430 |
56
-
57
 
58
 
59
  ### Reproduce the model
 
35
  Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d
36
 
37
  ```bash
38
+ lm_eval --model hf --model_args pretrained="Intel/gpt-j-6b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
39
  ```
40
 
41
 
42
+ | Metric | FP16 | INT4 |
 
43
  | -------------- | ------ | ------ |
44
+ | Avg. | 0.5039 | 0.5034 |
45
+ | mmlu | 0.2694 | 0.2793 |
46
+ | lambada_openai | 0.6831 | 0.6790 |
47
+ | hellaswag | 0.4953 | 0.4902 |
48
+ | winogrande | 0.6409 | 0.6401 |
49
+ | piqa | 0.7541 | 0.7465 |
50
+ | truthfulqa_mc1 | 0.2020 | 0.2179 |
51
+ | openbookqa | 0.2900 | 0.2900 |
52
+ | boolq | 0.6544 | 0.6554 |
53
+ | rte | 0.5451 | 0.5271 |
54
+ | arc_easy | 0.6692 | 0.6734 |
55
+ | arc_challenge | 0.3396 | 0.3387 |
56
 
57
 
58
  ### Reproduce the model