Create README.md

README.md

@@ -0,0 +1,25 @@
+# RWKV-6-World-1.6B-GGUF-Q4_K_M
+
+This repo contains the RWKV-6-World-1.6B-GGUF quantized with the latest llama.cpp(b3651).
+
+## How to run the model
+
+* Get the latest llama.cpp:
+```
+git clone https://github.com/ggerganov/llama.cpp
+```
+
+* Download the GGUF file to a new model folder in llama.cpp(just Q4_K_M for now, maybe I'll Q8 later):
+```
+cd llama.cpp
+mkdir model
+git clone https://huggingface.co/Lyte/RWKV-6-World-1.6B-GGUF/
+mv RWKV-6-World-1.6B-GGUF/RWKV-6-World-1.6B-GGUF-Q4_K_M.gguf model/
+rm -r RWKV-6-World-1.6B-GGUF
+```
+* For Windows other than git cloning the repo, you just create the "model" folder inside llama.cpp folder and go to the repo and download the model there.
+
+* Now to run the model, you can use the following command:
+```
+!./llama-cli -m ./model/RWKV-6-World-1.6B-GGUF-Q4_K_M.gguf --in-prefix "\nUser:" --in-suffix "\nAssistant:" --interactive-first -c 1024 -t 0.7 --top-k 40 --top-p 0.95 -n 64 -p "Assistant: Hello, what can i help you with today?\n" -r "User"
+```