|
--- |
|
base_model: meta-llama/Meta-Llama-3-8B |
|
inference: true |
|
model_type: llama |
|
pipeline_tag: text-generation |
|
tags: |
|
- sparse |
|
--- |
|
|
|
# SparseLlama-3-8B-pruned_50.2of4 |
|
|
|
This repo contains model files for a 2:4 (N:M) sparse [Meta-Llama-3-8B](meta-llama/Meta-Llama-3-8B) model pruned in one-shot with [SparseGPT](https://arxiv.org/abs/2301.00774), and then additionally retrained with the [SquareHead](https://arxiv.org/abs/2310.06927) knowledge distillation while maintaining the 2:4 sparsity mask. |
|
|
|
**Note:** This is still a work in progress and subject to change. We expect to release new weights with even better accuracy soon. |
|
|
|
## Running the model |
|
|
|
It can be run naively in transformers for testing purposes: |
|
```python |
|
# pip install transformers accelerate |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("nm-testing/SparseLlama-3-8B-pruned_50.2of4") |
|
model = AutoModelForCausalLM.from_pretrained("nm-testing/SparseLlama-3-8B-pruned_50.2of4", device_map="auto") |
|
|
|
input_text = "A poem about Machine Learning goes as follows:" |
|
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda") |
|
|
|
outputs = model.generate(**input_ids) |
|
print(tokenizer.decode(outputs[0])) |
|
``` |
|
|
|
To take advantage of the 2:4 sparsity present, install [nm-vllm](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage: |
|
```bash |
|
pip install nm-vllm[sparse] --extra-index-url https://pypi.neuralmagic.com/simple |
|
``` |
|
|
|
```python |
|
from vllm import LLM, SamplingParams |
|
|
|
model = LLM("nm-testing/SparseLlama-3-8B-pruned_50.2of4", sparsity="semi_structured_sparse_w16a16") |
|
|
|
prompt = "A poem about Machine Learning goes as follows:" |
|
sampling_params = SamplingParams(max_tokens=100, temperature=0) |
|
|
|
outputs = model.generate(prompt, sampling_params=sampling_params) |
|
print(outputs[0].outputs[0].text) |
|
``` |
|
|
|
## Evaluation Benchmark Results |
|
|
|
Model evaluation results obtained via [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) following the configuration of [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard). |
|
|
|
| Benchmark | Meta-Llama-3-8B | SparseLlama-3-8B-pruned_50.2of4<br>(this model) | |
|
|:----------------------------------------------:|:-----------:|:-----------------------------:| |
|
| [ARC-c](https://arxiv.org/abs/1911.01547)<br> 25-shot | 59.47% | 57.76% | |
|
| [MMLU](https://arxiv.org/abs/2009.03300)<br> 5-shot | 65.29% | 60.44% | |
|
| [HellaSwag](https://arxiv.org/abs/1905.07830)<br> 10-shot |82.14% | 79.97% | |
|
| [WinoGrande](https://arxiv.org/abs/1907.10641)<br> 5-shot |77.27% | 77.19% | |
|
| [GSM8K](https://arxiv.org/abs/2110.14168)<br> 5-shot | 44.81% | 47.92% | |
|
| [TruthfulQA](https://arxiv.org/abs/2109.07958)<br> 0-shot | 43.96% | 41.02% | |
|
| **Average<br>Accuracy** | **62.16%** | **60.72%** | |
|
| **Recovery** | **100%** | **97.68%** | |
|
|
|
|
|
Model evaluation results obtained via [Mosaic Eval Gauntlet](https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/EVAL_GAUNTLET.md) following the configuration of [Eval Gauntlet v0.3](https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/yamls/eval_gauntlet_v0.3.yaml). |
|
|
|
| Benchmark | Meta-Llama-3-8B | SparseLlama-3-8B-pruned_50.2of4<br>(this model) | |
|
|:------------------------:|:----------------:|:----------------------------------------------:| |
|
| World Knowledge | 58.08% | 54.61% | |
|
| Commonsense Reasoning | 47.66% | 47.62% | |
|
| Language Understanding | 71.13% | 67.58% | |
|
| Symbolic Problem Solving | 38.44% | 32.15% | |
|
| Reading Comprehension | 57.48% | 55.76% | |
|
| **Average Accuracy** | **54.70%** | **51.54%** | |
|
| **Recovery** | **100%** | **94.22%** | |
|
|
|
|
|
## Help |
|
|
|
For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ) |
|
|
|
## Acknowledgment |
|
|
|
This model is built with Meta Llama 3. For more details on its licence please check the model card of [Meta-Llama-3-8B](meta-llama/Meta-Llama-3-8B). |
