File size: 1,822 Bytes
362d372 dd1a343 88a477e 362d372 88a477e 362d372 88a477e b520c32 362d372 60feb0f b520c32 362d372 dd1a343 a29e824 dd1a343 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
---
license: llama3.1
tags:
- gguf
- llama3
pipeline_tag: text-generation
datasets:
- froggeric/imatrix
language:
- en
library_name: ggml
---
# Meta-Llama-3.1-405B-Instruct-GGUF
Low bit quantizations of Meta's Llama 3.1 405B Instruct model. Quantized from ollama q4_0 GGUF.
Quantized with llama.cpp [b3449](https://github.com/ggerganov/llama.cpp/releases/tag/b3449)
| Quant | Notes |
|-------------|--------------------------------------------|
| Q2_K | Suitable for general inference tasks |
| IQ2_XXS | Best for ultra-low memory footprint |
| IQ2_S | Optimized for small VRAM environments |
| Q3_K_M | Good balance between speed and accuracy |
| Q3_K_S | Faster inference with minor quality loss |
| Q3_K_L | High-quality with more VRAM requirement |
| Q4_K_M | Superior balance, suitable for production (although this is dequanted from q4_0, don't expect higher quality) |
| Q4_0 | Basic quantization, good for experimentation|
| Q4_K_S | Fast inference, efficient for scaling |
For higher quality quantizations (q4+), please refer to [nisten/meta-405b-instruct-cpu-optimized-gguf](https://huggingface.co/nisten/meta-405b-instruct-cpu-optimized-gguf).
Regarding the `smaug-bpe` tokenizer, this doesn't make a difference (they are identical). However, if you have concerns you can use the following command to set the `llama-bpe` tokenizer:
```
./gguf-py/scripts/gguf_new_metadata.py --pre-tokenizer "llama-bpe" Llama-3.1-405B-Instruct-old.gguf LLama-3.1-405B-Instruct-fixed.gguf
```
## imatrix
Generated from Q2_K quant.
imatrix calibration data: `groups_merged.txt` |