Commit
·
7401585
0
Parent(s):
GGUF model commit (made with llama.cpp commit 26d6076)
Browse files- .gitattributes +48 -0
- NSFW_13B_sft.F16.gguf +3 -0
- NSFW_13B_sft.Q2_K.gguf +3 -0
- NSFW_13B_sft.Q3_K_L.gguf +3 -0
- NSFW_13B_sft.Q3_K_M.gguf +3 -0
- NSFW_13B_sft.Q3_K_S.gguf +3 -0
- NSFW_13B_sft.Q4_0.gguf +3 -0
- NSFW_13B_sft.Q4_K_M.gguf +3 -0
- NSFW_13B_sft.Q4_K_S.gguf +3 -0
- NSFW_13B_sft.Q5_0.gguf +3 -0
- NSFW_13B_sft.Q5_K_M.gguf +3 -0
- NSFW_13B_sft.Q5_K_S.gguf +3 -0
- NSFW_13B_sft.Q6_K.gguf +3 -0
- NSFW_13B_sft.Q8_0.gguf +3 -0
- README.md +328 -0
.gitattributes
ADDED
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
36 |
+
NSFW_13B_sft.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
|
37 |
+
NSFW_13B_sft.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
38 |
+
NSFW_13B_sft.F16.gguf filter=lfs diff=lfs merge=lfs -text
|
39 |
+
NSFW_13B_sft.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
40 |
+
NSFW_13B_sft.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
41 |
+
NSFW_13B_sft.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
42 |
+
NSFW_13B_sft.Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
|
43 |
+
NSFW_13B_sft.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
|
44 |
+
NSFW_13B_sft.Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
|
45 |
+
NSFW_13B_sft.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
46 |
+
NSFW_13B_sft.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
|
47 |
+
NSFW_13B_sft.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
48 |
+
NSFW_13B_sft.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
NSFW_13B_sft.F16.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:45a7dda3b7e8e23144d245efd1920029e97e2a100a4267bb0ca3e7c715d2381a
|
3 |
+
size 26531994624
|
NSFW_13B_sft.Q2_K.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:39746578be70f4f89d0f381ab43c0d2909bec26511e829dc02a2e95a76d0c8e7
|
3 |
+
size 5562420288
|
NSFW_13B_sft.Q3_K_L.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:9a4dc400558cfd2b4f3f5abc46457d5e809b389223ac62a05d4dcefcd46fcc0e
|
3 |
+
size 7269735488
|
NSFW_13B_sft.Q3_K_M.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:a568b8f902ed5ca1e71e38ea6203ef16fc2e8bfe5f4504201e9aad6963c2cd72
|
3 |
+
size 6847601728
|
NSFW_13B_sft.Q3_K_S.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:bd4e5ad71beeead3bd9166dce12238d4548ca71e29642e5c13a49bf997fac6ea
|
3 |
+
size 6378445888
|
NSFW_13B_sft.Q4_0.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:d587e998b8ff0bc97ee34c21e2e7e41779a87b72e85fada90ced7b8fcc789c91
|
3 |
+
size 7548775488
|
NSFW_13B_sft.Q4_K_M.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:aef60c844d4bad128612a9d12fbe90d58250a7e4d4a1dbc37827f0f45e1bee95
|
3 |
+
size 8560487488
|
NSFW_13B_sft.Q4_K_S.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:ebbe8307fdc1c1a85d808b474194725420e9ab8e7a84611ee7d21ef3e23bf863
|
3 |
+
size 7934413888
|
NSFW_13B_sft.Q5_0.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:88dd512188a7797c84e98573c2c0c6410950f9f493b4fb8afb604081c8ce322d
|
3 |
+
size 9165876288
|
NSFW_13B_sft.Q5_K_M.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:0ecc01a8e9191a664fb5e569ff26f474c90096d24c2f5e7025e8edf01b6fdc2c
|
3 |
+
size 9849089088
|
NSFW_13B_sft.Q5_K_S.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:5cce984aa483d02b9b622259172ed728629113e767ed467b47159be7445f54ee
|
3 |
+
size 9341185088
|
NSFW_13B_sft.Q6_K.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:b53fc9ef9e98639dadf91e2a7ceaf2e0f85fade78ef167c67f706b010741fecc
|
3 |
+
size 11563367488
|
NSFW_13B_sft.Q8_0.gguf
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:dce9f6492e528d7d0217696f8460a8ea94d28e064131f93385aecb4a0187ad68
|
3 |
+
size 14096538688
|
README.md
ADDED
@@ -0,0 +1,328 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
base_model: zxbsmk/NSFW_13B_sft
|
3 |
+
inference: false
|
4 |
+
license: apache-2.0
|
5 |
+
model_creator: zxbsmk
|
6 |
+
model_name: NSFW 13B sft
|
7 |
+
model_type: baichuan
|
8 |
+
prompt_template: >
|
9 |
+
System: A chat between a curious human and an artificial intelligence
|
10 |
+
assistant. The assistant gives helpful, detailed, and polite answers to the
|
11 |
+
human's questions.
|
12 |
+
|
13 |
+
Human: {prompt}
|
14 |
+
|
15 |
+
Assistant:
|
16 |
+
quantized_by: mzwing
|
17 |
+
language:
|
18 |
+
- zh
|
19 |
+
tags:
|
20 |
+
- baichuan
|
21 |
+
- not-for-all-audiences
|
22 |
+
pipeline_tag: text-generation
|
23 |
+
datasets:
|
24 |
+
- zxbsmk/instruct_short_novel
|
25 |
+
---
|
26 |
+
|
27 |
+
# NSFW 13B sft - GGUF
|
28 |
+
|
29 |
+
- Model creator: [zxbsmk](https://huggingface.co/zxbsmk)
|
30 |
+
- Original model: [NSFW 13B sft](https://huggingface.co/zxbsmk/NSFW_13B_sft)
|
31 |
+
|
32 |
+
<!-- description start -->
|
33 |
+
|
34 |
+
## Description
|
35 |
+
|
36 |
+
This repo contains GGUF format model files for [zxbsmk's NSFW 13B sft](https://huggingface.co/zxbsmk/NSFW_13B_sft).
|
37 |
+
|
38 |
+
These files were quantised using hardware kindly provided by [Google Colab](https://colab.research.google.com/)(Free CPU Machine).
|
39 |
+
|
40 |
+
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mzwing/AI-related/blob/master/notebooks/NSFW_13B_sft-GGUF.ipynb)
|
41 |
+
|
42 |
+
You can also check it out easily in [my GitHub repo](https://github.com/mzwing/AI-related/blob/master/notebooks/NSFW_13B_sft-GGUF.ipynb).
|
43 |
+
|
44 |
+
<!-- description end -->
|
45 |
+
|
46 |
+
<!-- README_GGUF.md-about-gguf start -->
|
47 |
+
|
48 |
+
### About GGUF
|
49 |
+
|
50 |
+
GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
|
51 |
+
|
52 |
+
Here is an incomplate list of clients and libraries that are known to support GGUF:
|
53 |
+
|
54 |
+
* [llama.cpp](https://github.com/ggerganov/llama.cpp). The source project for GGUF. Offers a CLI and a server option.
|
55 |
+
* [text-generation-webui](https://github.com/oobabooga/text-generation-webui), the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
|
56 |
+
* [KoboldCpp](https://github.com/LostRuins/koboldcpp), a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
|
57 |
+
* [LM Studio](https://lmstudio.ai/), an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration.
|
58 |
+
* [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui), a great web UI with many interesting and unique features, including a full model library for easy model selection.
|
59 |
+
* [Faraday.dev](https://faraday.dev/), an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
|
60 |
+
* [ctransformers](https://github.com/marella/ctransformers), a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
|
61 |
+
* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
|
62 |
+
* [candle](https://github.com/huggingface/candle), a Rust ML framework with a focus on performance, including GPU support, and ease of use.
|
63 |
+
* [Nitro](https://nitro.jan.ai/), a fast, lightweight 3mb inference server to supercharge apps with local AI, and OpenAI-compatible API server.
|
64 |
+
|
65 |
+
<!-- README_GGUF.md-about-gguf end -->
|
66 |
+
|
67 |
+
<!-- repositories-available start -->
|
68 |
+
|
69 |
+
## Repositories available
|
70 |
+
|
71 |
+
* [2, 3, 4, 5, 6, 8, 16 and 32-bit GGUF models for CPU+GPU inference](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF)
|
72 |
+
* [zxbsmk's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/zxbsmk/NSFW_13B_sft)
|
73 |
+
|
74 |
+
<!-- repositories-available end -->
|
75 |
+
|
76 |
+
<!-- prompt-template start -->
|
77 |
+
|
78 |
+
## Prompt template: BLING
|
79 |
+
|
80 |
+
```
|
81 |
+
System: A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.
|
82 |
+
Human: {prompt}
|
83 |
+
Assistant:
|
84 |
+
|
85 |
+
```
|
86 |
+
|
87 |
+
<!-- prompt-template end -->
|
88 |
+
|
89 |
+
<!-- compatibility_gguf start -->
|
90 |
+
|
91 |
+
## Compatibility
|
92 |
+
|
93 |
+
These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit [d0cee0d](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221)
|
94 |
+
|
95 |
+
They are also compatible with many third party UIs and libraries - please see the list at the top of this README.
|
96 |
+
|
97 |
+
## Explanation of quantisation methods
|
98 |
+
|
99 |
+
<details>
|
100 |
+
<summary>Click to see details</summary>
|
101 |
+
|
102 |
+
The new methods available are:
|
103 |
+
|
104 |
+
* GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
|
105 |
+
* GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
|
106 |
+
* GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
|
107 |
+
* GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
|
108 |
+
* GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
|
109 |
+
|
110 |
+
Refer to the Provided Files table below to see what files use which methods, and how.
|
111 |
+
</details>
|
112 |
+
|
113 |
+
<!-- compatibility_gguf end -->
|
114 |
+
|
115 |
+
<!-- README_GGUF.md-provided-files start -->
|
116 |
+
|
117 |
+
## Provided files
|
118 |
+
|
119 |
+
| Name | Quant method | Bits | Size | Max RAM required | Use case |
|
120 |
+
| ---- | ---- | ---- | ---- | ---- | ----- |
|
121 |
+
| [NSFW_13B_sft.Q2_K.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q2_K.gguf) | Q2_K | 2 | 5.56 GB | untested yet | smallest, significant quality loss - not recommended for most purposes |
|
122 |
+
| [NSFW_13B_sft.Q3_K_S.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q3_K_S.gguf) | Q3_K_S | 3 | 6.38 GB | untested yet | very small, high quality loss |
|
123 |
+
| [NSFW_13B_sft.Q3_K_M.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q3_K_M.gguf) | Q3_K_M | 3 | 6.85 GB | untested yet | very small, high quality loss |
|
124 |
+
| [NSFW_13B_sft.Q3_K_L.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q3_K_L.gguf) | Q3_K_L | 3 | 7.27 GB | untested yet | small, substantial quality loss |
|
125 |
+
| [NSFW_13B_sft.Q4_0.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q4_0.gguf) | Q4_0 | 4 | 7.55 GB | untested yet | legacy; small, very high quality loss - prefer using Q3_K_M |
|
126 |
+
| [NSFW_13B_sft.Q4_K_S.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q4_K_S.gguf) | Q4_K_S | 4 | 7.93 GB | untested yet | small, greater quality loss |
|
127 |
+
| [NSFW_13B_sft.Q4_K_M.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q4_K_M.gguf) | Q4_K_M | 4 | 8.56 GB | untested yet | medium, balanced quality - recommended |
|
128 |
+
| [NSFW_13B_sft.Q5_0.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q5_0.gguf) | Q5_0 | 5 | 9.17 GB | untested yet | legacy; medium, balanced quality - prefer using Q4_K_M |
|
129 |
+
| [NSFW_13B_sft.Q5_K_S.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q5_K_S.gguf) | Q5_K_S | 5 | 9.34 GB | untested yet | large, low quality loss - recommended |
|
130 |
+
| [NSFW_13B_sft.Q5_K_M.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q5_K_M.gguf) | Q5_K_M | 5 | 9.85 GB | untested yet | large, very low quality loss - recommended |
|
131 |
+
| [NSFW_13B_sft.Q6_K.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q6_K.gguf) | Q6_K | 6 | 11.6 GB | untested yet | very large, extremely low quality loss |
|
132 |
+
| [NSFW_13B_sft.Q8_0.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.Q8_0.gguf) | Q8_0 | 8 | 14.1 GB | untested yet | very large, extremely low quality loss - not recommended |
|
133 |
+
| [NSFW_13B_sft.F16.gguf](https://huggingface.co/mzwing/NSFW_13B_sft-GGUF/blob/main/NSFW_13B_sft.F16.gguf) | F16 | 16 | 26.5 GB | untested yet | extremely large, extremely low quality loss - not recommended |
|
134 |
+
|
135 |
+
**Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
|
136 |
+
|
137 |
+
<!-- README_GGUF.md-provided-files end -->
|
138 |
+
|
139 |
+
<!-- README_GGUF.md-how-to-download start -->
|
140 |
+
|
141 |
+
## How to download GGUF files
|
142 |
+
|
143 |
+
**Note for manual downloaders:** You almost never want to clone the entire repo! Multiple different quantisation formats are provided, and most users only want to pick and download a single file.
|
144 |
+
|
145 |
+
The following clients/libraries will automatically download models for you, providing a list of available models to choose from:
|
146 |
+
|
147 |
+
* LM Studio
|
148 |
+
* LoLLMS Web UI
|
149 |
+
* Faraday.dev
|
150 |
+
|
151 |
+
### In `text-generation-webui`
|
152 |
+
|
153 |
+
Under Download Model, you can enter the model repo: `mzwing/NSFW_13B_sft-GGUF`, and below it, a specific filename to download, such as: `NSFW_13B_sft.Q4_K_M.gguf`.
|
154 |
+
|
155 |
+
Then click Download.
|
156 |
+
|
157 |
+
### On the command line, including multiple files at once
|
158 |
+
|
159 |
+
I recommend using the `huggingface-hub` Python library:
|
160 |
+
|
161 |
+
```shell
|
162 |
+
pip3 install huggingface-hub
|
163 |
+
```
|
164 |
+
|
165 |
+
Then you can download any individual model file to the current directory, at high speed, with a command like this:
|
166 |
+
|
167 |
+
```shell
|
168 |
+
huggingface-cli download mzwing/NSFW_13B_sft-GGUF NSFW_13B_sft.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
|
169 |
+
```
|
170 |
+
|
171 |
+
<details>
|
172 |
+
<summary>More advanced huggingface-cli download usage</summary>
|
173 |
+
|
174 |
+
You can also download multiple files at once with a pattern:
|
175 |
+
|
176 |
+
```shell
|
177 |
+
huggingface-cli download mzwing/NSFW_13B_sft-GGUF --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'
|
178 |
+
```
|
179 |
+
|
180 |
+
For more documentation on downloading with `huggingface-cli`, please see: [HF -> Hub Python Library -> Download files -> Download from the CLI](https://huggingface.co/docs/huggingface_hub/guides/download#download-from-the-cli).
|
181 |
+
|
182 |
+
To accelerate downloads on fast connections (1Gbit/s or higher), install `hf_transfer`:
|
183 |
+
|
184 |
+
```shell
|
185 |
+
pip3 install hf_transfer
|
186 |
+
```
|
187 |
+
|
188 |
+
And set environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`:
|
189 |
+
|
190 |
+
```shell
|
191 |
+
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download mzwing/NSFW_13B_sft-GGUF NSFW_13B_sft.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
|
192 |
+
```
|
193 |
+
|
194 |
+
Windows Command Line users: You can set the environment variable by running `set HF_HUB_ENABLE_HF_TRANSFER=1` before the download command.
|
195 |
+
</details>
|
196 |
+
<!-- README_GGUF.md-how-to-download end -->
|
197 |
+
|
198 |
+
<!-- README_GGUF.md-how-to-run start -->
|
199 |
+
## Example `llama.cpp` command
|
200 |
+
|
201 |
+
Make sure you are using `llama.cpp` from commit [d0cee0d](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221) or later.
|
202 |
+
|
203 |
+
```shell
|
204 |
+
./main -ngl 32 -m NSFW_13B_sft.Q4_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "System: A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\nHuman: {prompt}\nAssistant:"
|
205 |
+
```
|
206 |
+
|
207 |
+
Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
|
208 |
+
|
209 |
+
Change `-c 2048` to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically.
|
210 |
+
|
211 |
+
If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
|
212 |
+
|
213 |
+
For other parameters and how to use them, please refer to [the llama.cpp documentation](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md)
|
214 |
+
|
215 |
+
## How to run in `text-generation-webui`
|
216 |
+
|
217 |
+
Further instructions here: [text-generation-webui/docs/llama.cpp.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp.md).
|
218 |
+
|
219 |
+
## How to run from Python code
|
220 |
+
|
221 |
+
You can use GGUF models from Python using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) or [ctransformers](https://github.com/marella/ctransformers) libraries.
|
222 |
+
|
223 |
+
### How to load this model in Python code, using ctransformers
|
224 |
+
|
225 |
+
#### First install the package
|
226 |
+
|
227 |
+
Run one of the following commands, according to your system:
|
228 |
+
|
229 |
+
```shell
|
230 |
+
# Base ctransformers with no GPU acceleration
|
231 |
+
pip install ctransformers
|
232 |
+
# Or with CUDA GPU acceleration
|
233 |
+
pip install ctransformers[cuda]
|
234 |
+
# Or with AMD ROCm GPU acceleration (Linux only)
|
235 |
+
CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers
|
236 |
+
# Or with Metal GPU acceleration for macOS systems only
|
237 |
+
CT_METAL=1 pip install ctransformers --no-binary ctransformers
|
238 |
+
```
|
239 |
+
|
240 |
+
#### Simple ctransformers example code
|
241 |
+
|
242 |
+
```python
|
243 |
+
from ctransformers import AutoModelForCausalLM
|
244 |
+
|
245 |
+
# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
|
246 |
+
llm = AutoModelForCausalLM.from_pretrained("mzwing/NSFW_13B_sft-GGUF", model_file="NSFW_13B_sft.Q4_K_M.gguf", model_type="phi", gpu_layers=50)
|
247 |
+
|
248 |
+
print(llm("AI is going to"))
|
249 |
+
```
|
250 |
+
|
251 |
+
## How to use with LangChain
|
252 |
+
|
253 |
+
Here are guides on using llama-cpp-python and ctransformers with LangChain:
|
254 |
+
|
255 |
+
* [LangChain + llama-cpp-python](https://python.langchain.com/docs/integrations/llms/llamacpp)
|
256 |
+
* [LangChain + ctransformers](https://python.langchain.com/docs/integrations/providers/ctransformers)
|
257 |
+
|
258 |
+
<!-- README_GGUF.md-how-to-run end -->
|
259 |
+
|
260 |
+
<!-- footer start -->
|
261 |
+
<!-- 200823 -->
|
262 |
+
## Thanks, and how to contribute
|
263 |
+
|
264 |
+
Thanks to [Google Colab](https://colab.research.google.com/)! All the quantised models in this repo are done on the awesome platform. Thanks a lot!
|
265 |
+
|
266 |
+
Thanks to [llama.cpp](https://github.com/ggerganov/llama.cpp)! It inspired me to explore the inspiring AI field, thanks!
|
267 |
+
|
268 |
+
Thanks to [TheBloke](https://huggingface.co/TheBloke)! Everything in this repo is a reference to him.
|
269 |
+
|
270 |
+
You are welcome to create a **PullRequest**! Especially for the **RAM Usage**!
|
271 |
+
|
272 |
+
<!-- footer end -->
|
273 |
+
|
274 |
+
<!-- original-model-card start -->
|
275 |
+
|
276 |
+
A instruction-tuned model of https://huggingface.co/baichuan-inc/Baichuan-13B-Base
|
277 |
+
|
278 |
+
- Instruction-following datasets used: instruct_nsfw_cn
|
279 |
+
- Training framework: https://github.com/hiyouga/LLaMA-Efficient-Tuning
|
280 |
+
|
281 |
+
# Usage:
|
282 |
+
|
283 |
+
```python
|
284 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
|
285 |
+
tokenizer = AutoTokenizer.from_pretrained("zxbsmk/NSFW_13B_sft", trust_remote_code=True)
|
286 |
+
model = AutoModelForCausalLM.from_pretrained("zxbsmk/NSFW_13B_sft", trust_remote_code=True).cuda()
|
287 |
+
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
|
288 |
+
query = "续写。白洁已经动情了,肩带滑落露出香肩。"
|
289 |
+
inputs = tokenizer([query], return_tensors="pt")
|
290 |
+
inputs = inputs.to("cuda")
|
291 |
+
generate_ids = model.generate(**inputs, max_new_tokens=1024, streamer=streamer)
|
292 |
+
```
|
293 |
+
|
294 |
+
You could also alternatively launch a CLI demo by using the script in https://github.com/hiyouga/LLaMA-Efficient-Tuning
|
295 |
+
|
296 |
+
```bash
|
297 |
+
python src/cli_demo.py --template baichuan --model_name_or_path zxbsmk/NSFW_13B_sft
|
298 |
+
```
|
299 |
+
|
300 |
+
# Examples:
|
301 |
+
Omit part of results for better visualization.
|
302 |
+
|
303 |
+
| Question | NSFW_13B_sft | ChatGPT |
|
304 |
+
| -------- | ---------------- | ------- |
|
305 |
+
| 中国第三代战斗机的代表是什么? | 中国第三代战斗机的代表是歼-10��斗机。 |截至我所掌握的信息(截止2021年9月),中国第三代战斗机的代表是中国航空工业集团公司(AVIC)生产的"歼-20"战斗机。该战斗机是中国第三代隐形战斗机,也被称为"歼-20风暴",它是中国空军的先进战斗机,具备隐身特性、超音速巡航能力以及先进的雷达系统和电子设备。歼-20的研发和部署标志着中国在军事航空技术方面取得了重要进展。请注意,自那之后,可能已经有更多新型战斗机被研发或部署,因此建议查阅最新资料以获取更准确和最新的信息。|
|
306 |
+
| 解释这句话中每个词语的含义:今天下雨,我骑车差点摔倒,好在我一把把把把住了。 | 这句话中,“今天”指的是当前的时间,即雨天。“下雨”指的是天气阴沉,雨水落下。“我”指的是说话者,即骑车人。“骑车”指的是用自行车或其他交通工具出行。“差点”指的是几乎,几乎要摔倒。“摔倒”指的是身体失去平衡,摔在地上。“好在我”指的是幸亏有。“一把”指的是一下子。“把”指的是握住。“把住了”指的是握住车把,保持平衡。 | 今天:指代当天,即当前的日期。下雨:指天气现象,指雨水从天空降落。我:指说话人自己,第一人称代词。骑车:动词短语,表示骑着自行车。差点:几乎、险些,表示差一点就发生某种情况。摔倒:动词,意为因失去平衡而跌倒。好在:幸好、庆幸的意思,用来引出一个好的结果或意外。一把把把:这是一个错别字或打字错误,可能原本是“一把抓住”或类似的表达。考虑到语境,应该是指在骑车差点摔倒的情况下,通过抓住某物保住了平衡。抓住:动词短语,意为紧紧抓住某物,以保持稳定或防止掉落。综合起来,这句话的意思是:今天因为下雨,我骑车时差点摔倒,幸好我抓住了某物,避免了摔倒。 |
|
307 |
+
| 续写。白洁已经动情了,肩带滑落露出香肩。 | 白洁低下头,将秀发从肩头挽起,露出娇俏的耳垂,戴着个银制的耳钉,是叶子送给她的生日礼物,上面有个心型的装饰,是叶子亲手在心型装饰上刻上了白洁的名字。白洁没有穿胸罩,一对丰满的乳房随着她的动作轻轻颤动,滑落的肩带垂落在乳房上,更平添了一份诱惑... | 她静静地坐在窗边,微风轻拂着她的长发,将她的芳香散发到整个房间。白洁的心情久久不能平静,她的内心充满了对他的思念和渴望。她闭上眼睛,回想起他们之间的点点滴滴,那些温柔的瞬间在她的脑海中浮现... |
|
308 |
+
| 续写一下片段。王健已经忍耐不住了,一把搂住陆春兰,狠狠地揉捏她的巨乳。 | “嗯......王校长......轻点,哎呀......”陆春兰痛呼一声,连忙推开王健,一边赶紧拉开自己的上衣,露出雪白的胸脯,擦干乳房上的汗水,一边瞪了王健一眼,“都是你,弄得我这么黏糊糊的。”... | 很抱歉,但我无法为你提供所请求的续写。 |
|
309 |
+
|
310 |
+
# Evaluation
|
311 |
+
Comparision between Baichuan-13B-Chat and NSFW_13B_sft.
|
312 |
+
(Zero-shot [CMMLU](https://github.com/haonan-li/CMMLU))
|
313 |
+
|
314 |
+
| Score | NSFW_13B_sft | Baichuan-13B-Chat | ChatGPT |
|
315 |
+
| -------- | ---------------- | ------- |------- |
|
316 |
+
| STEM | 37.73 | 37.00 |**44.80** |
|
317 |
+
| Humanities | **54.85** | 53.74 |53.61 |
|
318 |
+
| Social Sciences | **55.55** | 52.77 |54.22 |
|
319 |
+
| Other | 53.47 | 52.31 |**59.95** |
|
320 |
+
| China specific | **51.84** | 50.55 |49.74 |
|
321 |
+
| Overall | 50.42 | 48.86 |**53.22** |
|
322 |
+
|
323 |
+
(By the way, Baichuan-13B-Chat gets 50.43 with one-shot which seems much better than 48.86 with zero-shot.)
|
324 |
+
|
325 |
+
# Contact Us
|
326 |
+
Join group via https://t.me/+JbovpBG6-gBiNDI1
|
327 |
+
|
328 |
+
<!-- original-model-card end -->
|