File size: 757 Bytes
e907dc4 197ddee e907dc4 ada6f20 e907dc4 7fe1357 57875eb e907dc4 ada6f20 e907dc4 ada6f20 e907dc4 197ddee |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
---
library_name: transformers
license: gemma
---
# Model Card for Model ID
### Model Description
<!-- Provide a longer summary of what this model is. -->
This is a model with 4-bit quantization using the following code.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import BitsAndBytesConfig
import torch
model_name = "google/gemma-2-27b"
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=quantization_config,
device_map="auto"
)
``` |