Update README.md
Browse files
README.md
CHANGED
@@ -35,25 +35,24 @@ print(tokenizer.decode(model.generate(**tokenizer("There is a girl who likes adv
|
|
35 |
Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d
|
36 |
|
37 |
```bash
|
38 |
-
lm_eval --model hf --model_args pretrained="Intel/gpt-j-6b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,mmlu --batch_size 32
|
39 |
```
|
40 |
|
41 |
|
42 |
-
|
43 |
-
| Metric | BF16 | INT4 |
|
44 |
| -------------- | ------ | ------ |
|
45 |
-
| Avg. | 0. | 0.
|
46 |
-
| mmlu | 0. | 0.
|
47 |
-
| lambada_openai | 0. | 0.
|
48 |
-
| hellaswag | 0. | 0.
|
49 |
-
| winogrande | 0. | 0.
|
50 |
-
| piqa | 0. | 0.
|
51 |
-
| truthfulqa_mc1 | 0. | 0.
|
52 |
-
| openbookqa | 0. | 0.
|
53 |
-
| boolq | 0. | 0.
|
54 |
-
|
|
55 |
-
|
|
56 |
-
|
57 |
|
58 |
|
59 |
### Reproduce the model
|
|
|
35 |
Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d
|
36 |
|
37 |
```bash
|
38 |
+
lm_eval --model hf --model_args pretrained="Intel/gpt-j-6b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
|
39 |
```
|
40 |
|
41 |
|
42 |
+
| Metric | FP16 | INT4 |
|
|
|
43 |
| -------------- | ------ | ------ |
|
44 |
+
| Avg. | 0.5039 | 0.5034 |
|
45 |
+
| mmlu | 0.2694 | 0.2793 |
|
46 |
+
| lambada_openai | 0.6831 | 0.6790 |
|
47 |
+
| hellaswag | 0.4953 | 0.4902 |
|
48 |
+
| winogrande | 0.6409 | 0.6401 |
|
49 |
+
| piqa | 0.7541 | 0.7465 |
|
50 |
+
| truthfulqa_mc1 | 0.2020 | 0.2179 |
|
51 |
+
| openbookqa | 0.2900 | 0.2900 |
|
52 |
+
| boolq | 0.6544 | 0.6554 |
|
53 |
+
| rte | 0.5451 | 0.5271 |
|
54 |
+
| arc_easy | 0.6692 | 0.6734 |
|
55 |
+
| arc_challenge | 0.3396 | 0.3387 |
|
56 |
|
57 |
|
58 |
### Reproduce the model
|