added example in README

README.md

@@ -8,6 +8,44 @@ license: apache-2.0
 
 # LLaVA Model Card
 
+## SGLang
+This contains the necessary files to run LLaVA-1.6 34B on SGLang. You can run the server with the following command:
+
+`python -m sglang.launch_server --model-path dillonlaird/hf-llava-v1.6-34b --port 30000`
+
+There seems to be issues with the chat formatting when using the sglang interface so I recommend querying the server directly and formatting the string yourself:
+
+```python
+import requests
+from transformers import AutoTokenizer
+
+
+def generate(image_path: str, prompt: str, tokenizer):
+    chat = [
+        {"role": "system", "content": "Answer the question."},
+        {"role": "user", "content": "<image>\n" + prompt},
+    ]
+    chat_str = tokenizer.apply_chat_template(chat, tokenize=False)
+    chat_str += "<|img_start|>assistant\n"
+    sampling_params = {"temperature": 0.2, "max_new_tokens": 1536}
+    res = requests.post(
+        "http://localhost:30000/generate",
+        json={
+            "text": chat_str,
+            "image_data": image_path,
+            "sampling_params": sampling_params,
+        },
+    )
+    return res.json()["text"]
+
+
+if __name__ == "__main__":
+    tokenizer = AutoTokenizer.from_pretrained("liuhaotian/llava-v1.6-34b")
+    image_path = "path/to/image.jpg"
+    prompt = "What is the name of the mountain?"
+    desc = generate(image_path, prompt, tokenizer)
+```
+
 ## Model details
 
 **Model type:**
@@ -42,4 +80,4 @@ The primary intended users of the model are researchers and hobbyists in compute
 - 40K ShareGPT data.
 
 ## Evaluation dataset
-A collection of 12 benchmarks, including 5 academic VQA benchmarks and 7 recent benchmarks specifically proposed for instruction-following LMMs.
+A collection of 12 benchmarks, including 5 academic VQA benchmarks and 7 recent benchmarks specifically proposed for instruction-following LMMs.