Initial GGML model commit

README.md

@@ -0,0 +1,169 @@
+---
+inference: false
+license: other
+---
+
+<!-- header start -->
+<div style="width: 100%;">
+    <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
+</div>
+<div style="display: flex; justify-content: space-between; width: 100%;">
+    <div style="display: flex; flex-direction: column; align-items: flex-start;">
+        <p><a href="https://discord.gg/theblokeai">Chat & support: my new Discord server</a></p>
+    </div>
+    <div style="display: flex; flex-direction: column; align-items: flex-end;">
+        <p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
+    </div>
+</div>
+<!-- header end -->
+
+# Open BMB's UltraLM 13B GGML
+
+These files are GGML format model files for [Open BMB's UltraLM 13B](https://huggingface.co/openbmb/UltraLM-13b).
+
+GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
+* [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
+* [KoboldCpp](https://github.com/LostRuins/koboldcpp)
+* [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui)
+* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
+* [ctransformers](https://github.com/marella/ctransformers)
+
+## Repositories available
+
+* [4-bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/UltraLM-13B-GPTQ)
+* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/UltraLM-13B-GGML)
+* [Unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/TheBloke/UltraLM-13B-fp16)
+
+<!-- compatibility_ggml start -->
+## Compatibility
+
+### Original llama.cpp quant methods: `q4_0, q4_1, q5_0, q5_1, q8_0`
+
+I have quantized these 'original' quantisation methods using an older version of llama.cpp so that they remain compatible with llama.cpp as of May 19th, commit `2d5db48`.
+
+These are guaranteed to be compatbile with any UIs, tools and libraries released since late May.
+
+### New k-quant methods: `q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K`
+
+These new quantisation methods are compatible with llama.cpp as of June 6th, commit `2d43387`.
+
+They are now also compatible with recent releases of text-generation-webui, KoboldCpp, llama-cpp-python and ctransformers. Other tools and libraries may or may not be compatible - check their documentation if in doubt.
+
+## Explanation of the new k-quant methods
+
+The new methods available are:
+* GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
+* GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
+* GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
+* GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
+* GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
+* GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type.
+
+Refer to the Provided Files table below to see what files use which methods, and how.
+<!-- compatibility_ggml end -->
+
+## Provided files
+| Name | Quant method | Bits | Size | Max RAM required | Use case |
+| ---- | ---- | ---- | ---- | ---- | ----- |
+| ultralm-13b.ggmlv3.q4_0.bin | q4_0 | 4 | 7.32 GB | 9.82 GB | Original llama.cpp quant method, 4-bit. |
+| ultralm-13b.ggmlv3.q4_1.bin | q4_1 | 4 | 8.14 GB | 10.64 GB | Original llama.cpp quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
+| ultralm-13b.ggmlv3.q5_0.bin | q5_0 | 5 | 8.95 GB | 11.45 GB | Original llama.cpp quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
+| ultralm-13b.ggmlv3.q5_1.bin | q5_1 | 5 | 9.76 GB | 12.26 GB | Original llama.cpp quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
+| ultralm-13b.ggmlv3.q8_0.bin | q8_0 | 8 | 13.83 GB | 16.33 GB | Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
+
+**Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
+
+## How to run in `llama.cpp`
+
+I use the following command line; adjust for your tastes and needs:
+
+```
+./main -t 10 -ngl 32 -m ultralm-13b.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"
+```
+If you're able to use full GPU offloading, you should use `-t 1` to get best performance.
+
+If not able to fully offload to GPU, you should use more cores. Change `-t 10` to the number of physical CPU cores you have, or a lower number depending on what gives best performance.
+
+Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
+
+If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
+
+## How to run in `text-generation-webui`
+
+Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md).
+
+<!-- footer start -->
+## Discord
+
+For further support, and discussions on these models and AI in general, join us at:
+
+[TheBloke AI's Discord server](https://discord.gg/theblokeai)
+
+## Thanks, and how to contribute.
+
+Thanks to the [chirper.ai](https://chirper.ai) team!
+
+I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
+
+If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.
+
+Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.
+
+* Patreon: https://patreon.com/TheBlokeAI
+* Ko-Fi: https://ko-fi.com/TheBlokeAI
+
+**Special thanks to**: Luke from CarbonQuill, Aemon Algiz, Dmitriy Samsonov.
+
+**Patreon special mentions**: zynix, ya boyyy, Trenton Dambrowitz, Imad Khwaja, Alps Aficionado, chris gileta, John Detwiler, Willem Michiel, RoA, Mano Prime, Rainer Wilmers, Fred von Graf, Matthew Berman, Ghost , Nathan LeClaire, Iucharbius , Ai Maven, Illia Dulskyi, Joseph William Delisle, Space Cruiser, Lone Striker, Karl Bernard, Eugene Pentland, Greatston Gnanesh, Jonathan Leane, Randy H, Pierre Kircher, Willian Hasse, Stephen Murray, Alex , terasurfer , Edmond Seymore, Oscar Rangel, Luke Pendergrass, Asp the Wyvern, Junyu Yang, David Flickinger, Luke, Spiking Neurons AB, subjectnull, Pyrater, Nikolai Manek, senxiiz, Ajan Kanaga, Johann-Peter Hartmann, Artur Olbinski, Kevin Schuppel, Derek Yates, Kalila, K, Talal Aujan, Khalefa Al-Ahmad, Gabriel Puliatti, John Villwock, WelcomeToTheClub, Daniel P. Andersen, Preetika Verma, Deep Realms, Fen Risland, trip7s trip, webtim, Sean Connelly, Michael Levine, Chris McCloskey, biorpg, vamX, Viktor Bowallius, Cory Kujawski.
+
+Thank you to all my generous patrons and donaters!
+
+<!-- footer end -->
+
+# Original model card: Open BMB's UltraLM 13B
+
+# UltraLM-13b
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+This is UltraLM-13b delta weights, a chat language model trained upon [UltraChat](https://github.com/thunlp/UltraChat)
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+The model is fine-tuned based on LLaMA-13b with a multi-turn chat-format template as below
+
+```
+User: instruction 1<eos_token>
+Assistant: response 1<eos_token>
+User: instruction 2<eos_token>
+Assistant: response 2<eos_token>
+...
+```
+
+- **License:** UltraLM is based on LLaMA and should be used under LLaMA's [model license](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md).
+- **Finetuned from model:** LLaMA-13b
+- **Finetuned on data:** [UltraChat](https://github.com/thunlp/UltraChat)
+
+### Model Sources
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [UltraChat](https://github.com/thunlp/UltraChat)
+- **Paper:** [arxiv](https://arxiv.org/abs/2305.14233)
+- **Demo:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+To use this model, you need to [recover](https://github.com/thunlp/UltraChat/tree/main/UltraLM) the full model from the delta weights and perform inference following the template below:
+
+```
+[Optional]User: system prompt<eos_token>
+User: user input<eos_token>
+Assistant: 
+```