Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,117 @@
+---
+language:
+  - en
+  - fr
+  - de
+  - es
+  - it
+  - pt
+  - ja
+  - ko
+  - zh
+  - ar
+  - el
+  - fa
+  - pl
+  - id
+  - cs
+  - he
+  - hi
+  - nl
+  - ro
+  - ru
+  - tr
+  - uk
+  - vi
+license: cc-by-nc-4.0
+library_name: transformers
+tags:
+- cohere
+- pytorch
+- awq
+model_name: aya-expanse-8b-awq-4bit
+base_model: CohereForAI/aya-expanse-8b
+inference: false
+model_creator: Cohere For AI
+pipeline_tag: text-generation
+quantized_by: kevinbazira
+---
+
+# aya-expanse-8b-awq-4bit
+
+This repository contains a quantized version of the `CohereForAI/aya-expanse-8b` model using the [AWQ](https://huggingface.co/docs/transformers/en/quantization/awq) method in 4-bit precision.
+
+## Model Summary
+
+- **Quantized Model**: [kevinbazira/aya-expanse-8b-awq-4bit](https://huggingface.co/kevinbazira/aya-expanse-8b-awq-4bit)
+- **Quantization Method**: [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://arxiv.org/pdf/2306.00978)
+- **AWQ Version**: [GEMM](https://github.com/casper-hansen/AutoAWQ/tree/f1abb8ef8e261db78eb6c603f691801797fbb293?tab=readme-ov-file#int4-gemm-vs-int4-gemv-vs-fp16)
+- **Precision**: 4-bit
+- **Original Model**: [CohereForAI/aya-expanse-8b](https://huggingface.co/CohereForAI/aya-expanse-8b)
+
+## How to Use the Quantized Model
+
+### 1. Install the necessary packages
+
+Before using the quantized model, please ensure your environment has:
+- [AutoAWQ_kernels](https://github.com/casper-hansen/AutoAWQ_kernels)
+- [AutoAWQ](https://github.com/casper-hansen/AutoAWQ)
+
+### 2. Run inference
+Load and use the quantized model as shown below in Python:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, AwqConfig
+
+# Set up device
+device = torch.device('cuda:1') # Remember to use the correct device here
+
+# Load model and tokenizer
+model_name = "kevinbazira/aya-expanse-8b-awq-4bit"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+quantization_config = AwqConfig(version="exllama")
+model = AutoModelForCausalLM.from_pretrained(
+    model_name, 
+    device_map={"": device.index},
+    quantization_config=quantization_config
+)
+
+# Prepare input
+# https://huggingface.co/docs/transformers/en/pad_truncation
+input_text = "Add your prompt here."
+inputs = tokenizer(input_text, return_tensors="pt", truncation=True, padding="max_length", max_length=64)
+inputs = {key: value.to(device) for key, value in inputs.items()}
+
+# Perform text generation 
+# https://huggingface.co/docs/transformers/en/main_classes/text_generation
+outputs = model.generate(
+    **inputs,
+    num_return_sequences=1,
+    min_new_tokens=64,
+    max_new_tokens=64,
+    do_sample=False,
+    use_cache=True,
+    num_beams=1
+)
+
+# Decode and print the output
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Benchmark Results
+
+To evaluate the performance of the quantized model, we run benchmarks using the Hugging Face [Optimum Benchmark](https://github.com/huggingface/optimum-benchmark/tree/7cec62e016d76fe612308e4c2c074fc7f09289fd) tool on an AMD MI200 GPU with ROCm 6.1 and below are the results:
+
+### Unquantized Model Results:
+<img src="unquantized-model-results.png" alt="Unquantized Model Results" style="width: 100%; object-fit: cover; display: block;">
+
+### AWQ Quantized Model Results:
+<img src="awq-quantized-model-results.png" alt="AWQ Quantized Model Results" style="width: 100%; object-fit: cover; display: block;">
+
+These results show that the AWQ quantized model offers significant speed advantages during critical inference stages (decode and per-token), outweighing the higher latencies encountered during the load and prefill phases. For deployment scenarios where inference speed is paramount, you can preload the quantized model to eliminate initial latency concerns.
+
+## More Information
+
+- **Original Model**: For details about the original model's architecture, training dataset, and performance, please visit the CohereForAI [aya-expanse-8b model card](https://huggingface.co/CohereForAI/aya-expanse-8b).
+- **Support or inquiries**: If you run into any issues or have questions about the quantized model, feel free to reach me via email:`[email protected]`. I'll be happy to help!