File size: 1,954 Bytes
62173f4 6000607 62173f4 6000607 467e149 62173f4 8cc9887 62173f4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 |
---
base_model: facebook/opt-13b
language:
- en
license: other
model_name: opt-13b
pipeline_tag: text-generation
inference: false
model_creator: facebook
model_type: opt
quantized_by: iproskurina
tags:
- gptq
- 4-bit
base_model_relation: quantized
---
<img src="https://cdn-uploads.huggingface.co/production/uploads/629a3dbcd496c6dcdebf41cc/t-6kpqFpEYJPT6zmvnm49.png" width="200" />
# OPT-13B - GPTQ
- Model creator: [Meta AI](https://huggingface.co/facebook)
- Original model: [OPT-13B](https://huggingface.co/facebook/opt-13b)
The model published in this repo was quantized to 4bit using [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ).
**Quantization details**
**All quantization parameters were taken from [GPTQ paper](https://arxiv.org/abs/2210.17323).**
GPTQ calibration data consisted of 128 random 2048 token segments from the [C4 dataset](https://huggingface.co/datasets/c4).
The grouping size used for quantization is equal to 128.
## How to use this GPTQ model from Python code
### Install the necessary packages
```shell
pip install accelerate==0.26.1 datasets==2.16.1 dill==0.3.7 gekko==1.0.6 multiprocess==0.70.15 peft==0.7.1 rouge==1.0.1 sentencepiece==0.1.99
git clone https://github.com/upunaprosk/AutoGPTQ
cd AutoGPTQ
pip install -v .
```
Recommended transformers version: 4.35.2.
### You can then use the following code
```python
from transformers import AutoTokenizer, TextGenerationPipeline,AutoModelForCausalLM
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
pretrained_model_dir = "iproskurina/opt-13b-gptq-4bit"
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True)
model = AutoGPTQForCausalLM.from_quantized(pretrained_model_dir, device="cuda:0", model_basename="model")
pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer)
print(pipeline("auto-gptq is")[0]["generated_text"])
```
[**LICENSE**](https://huggingface.co/facebook/opt-13b/blob/main/LICENSE.md)
|