|
--- |
|
license: llama2 |
|
language: |
|
- en |
|
pipeline_tag: conversational |
|
--- |
|
Another EXL2 version of AlpinDale's https://huggingface.co/alpindale/goliath-120b this one being at 2.37BPW. |
|
|
|
[2.64BPW](https://huggingface.co/LavaPlanet/Goliath120B-exl2-2.64bpw) |
|
|
|
Pippa llama2 Chat was used as the calibration dataset. |
|
|
|
Can be run on two RTX 3090s w/ 24GB vram each. |
|
|
|
Assuming Windows overhead, the following figures should be more or less close enough for estimation of your own use. |
|
```yaml |
|
2.37BPW @ 4096 ctx |
|
empty ctx |
|
GPU split: 16/24 |
|
GPU1: 17.4/24GB |
|
GPU2: 19.5/24GB |
|
11~ tk/s |
|
3000+ ctx |
|
8~-12 tk/s |
|
``` |
