legraphista commited on
Commit
4f79769
·
verified ·
1 Parent(s): 84c580b

add: chat template and inference example

Browse files
Files changed (1) hide show
  1. README.md +31 -0
README.md CHANGED
@@ -78,6 +78,37 @@ huggingface-cli download legraphista/DeepSeek-V2-Lite-Chat-IMat-GGUF --include "
78
  # see FAQ for merging GGUF's
79
  ```
80
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
  ## FAQ
82
 
83
  ### Why is the IMatrix not applied everywhere?
 
78
  # see FAQ for merging GGUF's
79
  ```
80
 
81
+ ---
82
+
83
+ ## Inference
84
+
85
+ ### Simple chat template
86
+ ```
87
+ <|begin▁of▁sentence|>User: {user_message_1}
88
+
89
+ Assistant: {assistant_message_1}<|end▁of▁sentence|>User: {user_message_2}
90
+
91
+ Assistant:
92
+ ```
93
+
94
+ ### Chat template with system prompt
95
+ ```
96
+ <|begin▁of▁sentence|>{system_message}
97
+
98
+ User: {user_message_1}
99
+
100
+ Assistant: {assistant_message_1}<|end▁of▁sentence|>User: {user_message_2}
101
+
102
+ Assistant:
103
+ ```
104
+
105
+ ### Llama.cpp
106
+ ```
107
+ llama.cpp/main -m DeepSeek-V2-Lite-Chat.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"
108
+ ```
109
+
110
+ ---
111
+
112
  ## FAQ
113
 
114
  ### Why is the IMatrix not applied everywhere?