|
--- |
|
license: unknown |
|
--- |
|
|
|
[ehartford/WizardLM-7B-Uncensored](https://huggingface.co/ehartford/WizardLM-7B-Uncensored) quantized to **8bit GPTQ** with act order + true sequential, no group size. |
|
|
|
*For most uses this probably isn't what you want.* \ |
|
*For 4bit with no act order or compatibility with `old-cuda` (text-generation-webui default) see [TheBloke/WizardLM-7B-uncensored-GPTQ](https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GPTQ)* |
|
|
|
Quantized using AutoGPTQ with the following config: |
|
```python |
|
config: dict = dict( |
|
quantize_config=dict(bits=8, desc_act=True, true_sequential=True, model_file_base_name='WizardLM-7B-Uncensored'), |
|
use_safetensors=True |
|
) |
|
``` |
|
See `quantize.py` for the full script. |
|
|
|
Tested for compatibility with: |
|
- WSL with GPTQ-for-Llama `triton` branch. |
|
- Windows with AutoGPTQ on `cuda` (triton deselected) |
|
|
|
AutoGPTQ loader should read configuration from `quantize_config.json`.\ |
|
For GPTQ-for-Llama use the following configuration when loading:\ |
|
wbits: 8\ |
|
groupsize: None\ |
|
model_type: llama |
|
|
|
|
