File size: 1,979 Bytes
b31244a 6463fd2 58bcc76 6463fd2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 |
---
tags:
- experimental
- testing
- gguf
- roleplay
- quantized
- mistral
- text-generation-inference
---
**These are quants for an experimental model.**
"Q4_K_M", "Q4_K_S", "IQ4_XS", "Q5_K_M", "Q5_K_S",
"Q6_K", "Q8_0", "IQ3_M", "IQ3_S", "IQ3_XXS"
Original model weights: <br> https://huggingface.co/Nitral-AI/Eris_PrimeV4-Vision-7B
# Vision/multimodal capabilities:
<details><summary>
Click here to see how this would work in practice in a roleplay chat.
</summary>
</details><br>
<details><summary>
Click here to see what your SillyTavern Image Captions extension settings should look like.
</summary>
</details><br>
**If you want to use vision functionality:**
* Make sure you are using the latest version of [KoboldCpp](https://github.com/LostRuins/koboldcpp).
To use the multimodal capabilities of this model, such as **vision**, you also need to load the specified **mmproj** file, you can get it [here](https://huggingface.co/cjpais/llava-1.6-mistral-7b-gguf/blob/main/mmproj-model-f16.gguf), it's also hosted in this repository inside the **mmproj** folder.
* You can load the **mmproj** by using the corresponding section in the interface:
* For CLI users, you can load the **mmproj file** by adding the respective flag to your usual command:
```
--mmproj your-mmproj-file.gguf
```
# Quantization information:
**Steps performed:**
```
Base⇢ GGUF(F16)⇢ Imatrix-Data(F16)⇢ GGUF(Imatrix-Quants)
```
*Using the latest llama.cpp at the time.* |