Upload README.md

README.md

@@ -12,39 +12,67 @@ tags:
 ---
 Quantizations of https://huggingface.co/microsoft/Phi-3-mini-128k-instruct
 
+**Requantized and reuploaded!** GGUFs for the latest Phi-3 mini model (July 2024 update), which includes::
+* Significantly increased code understanding in Python, C++, Rust, and Typescript.
+* Enhanced post-training for better-structured output.
+* Improved multi-turn instruction following.
+* Support for <|system|> tag.
+* Improved reasoning and long-context understanding.
+
+### Inference Clients/UIs
+* [llama.cpp](https://github.com/ggerganov/llama.cpp)
+* [JanAI](https://github.com/janhq/jan)
+* [KoboldCPP](https://github.com/LostRuins/koboldcpp)
+* [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
+* [ollama](https://github.com/ollama/ollama)
+
+---
+
 # From original readme
 
 ## How to Use
 
-Phi-3 Mini-128K-Instruct has been integrated in the development version (4.41.0.dev0) of `transformers`. Until the official version is released through `pip`, ensure that you are doing one of the following:
-
-* When loading the model, ensure that `trust_remote_code=True` is passed as an argument of the `from_pretrained()` function.
+Phi-3 Mini-4K-Instruct has been integrated in the `4.41.2` version of `transformers`. The current `transformers` version can be verified with: `pip list | grep transformers`.
 
-* Update your local `transformers` to the development version: `pip uninstall -y transformers && pip install git+https://github.com/huggingface/transformers`. The previous command is an alternative to cloning and installing from the source.
+Examples of required packages:
+```
+flash_attn==2.5.8
+torch==2.3.1
+accelerate==0.31.0
+transformers==4.41.2
+```
 
-The current `transformers` version can be verified with: `pip list | grep transformers`.
+Phi-3 Mini-4K-Instruct is also available in [Azure AI Studio](https://aka.ms/try-phi3)
 
 ### Tokenizer
 
-Phi-3 Mini-128K-Instruct supports a vocabulary size of up to `32064` tokens. The [tokenizer files](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/blob/main/added_tokens.json) already provide placeholder tokens that can be used for downstream fine-tuning, but they can also be extended up to the model's vocabulary size.
+Phi-3 Mini-4K-Instruct supports a vocabulary size of up to `32064` tokens. The [tokenizer files](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/blob/main/added_tokens.json) already provide placeholder tokens that can be used for downstream fine-tuning, but they can also be extended up to the model's vocabulary size.
 
 ### Chat Format
 
-Given the nature of the training data, the Phi-3 Mini-128K-Instruct model is best suited for prompts using the chat format as follows.
+Given the nature of the training data, the Phi-3 Mini-4K-Instruct model is best suited for prompts using the chat format as follows. 
 You can provide the prompt as a question with a generic template as follow:
 ```markdown
-<|user|>\nQuestion<|end|>\n<|assistant|>
+<|system|>
+You are a helpful assistant.<|end|>
+<|user|>
+Question?<|end|>
+<|assistant|>
 ```
+
 For example:
 ```markdown
+<|system|>
+You are a helpful assistant.<|end|>
 <|user|>
 How to explain Internet for a medieval knight?<|end|>
-<|assistant|>
+<|assistant|> 
 ```
-
-where the model generates the text after `<|assistant|>`. In case of few-shots prompt, the prompt can be formatted as the following:
+where the model generates the text after `<|assistant|>` . In case of few-shots prompt, the prompt can be formatted as the following:
 
 ```markdown
+<|system|>
+You are a helpful travel assistant.<|end|>
 <|user|>
 I am going to Paris, what should I see?<|end|>
 <|assistant|>
@@ -59,40 +87,39 @@ What is so great about #1?<|end|>
 This code snippets show how to get quickly started with running the model on a GPU:
 
 ```python
-import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
-
-torch.random.manual_seed(0)
-
-model = AutoModelForCausalLM.from_pretrained(
-    "microsoft/Phi-3-mini-128k-instruct", 
-    device_map="cuda", 
-    torch_dtype="auto", 
-    trust_remote_code=True, 
-)
-tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-128k-instruct")
-
-messages = [
-    {"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"},
-    {"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."},
-    {"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"},
-]
-
-pipe = pipeline(
-    "text-generation",
-    model=model,
-    tokenizer=tokenizer,
-)
-
-generation_args = {
-    "max_new_tokens": 500,
-    "return_full_text": False,
-    "temperature": 0.0,
-    "do_sample": False,
-}
-
-output = pipe(messages, **generation_args)
-print(output[0]['generated_text'])
-```
-
-*Some applications/frameworks might not include a BOS token (`<s>`) at the start of the conversation. Please ensure that it is included since it provides more reliable results.*
+import torch 
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline 
+
+torch.random.manual_seed(0) 
+model = AutoModelForCausalLM.from_pretrained( 
+    "microsoft/Phi-3-mini-4k-instruct",  
+    device_map="cuda",  
+    torch_dtype="auto",  
+    trust_remote_code=True,  
+) 
+
+tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct") 
+
+messages = [ 
+    {"role": "system", "content": "You are a helpful AI assistant."}, 
+    {"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"}, 
+    {"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."}, 
+    {"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"}, 
+] 
+
+pipe = pipeline( 
+    "text-generation", 
+    model=model, 
+    tokenizer=tokenizer, 
+) 
+
+generation_args = { 
+    "max_new_tokens": 500, 
+    "return_full_text": False, 
+    "temperature": 0.0, 
+    "do_sample": False, 
+} 
+
+output = pipe(messages, **generation_args) 
+print(output[0]['generated_text']) 
+```