All responses come back as "!!!!!..." repeated like 100 times
#10
by
jamie-de
- opened
I've got this model up on an inference endpoint, and I've tried both a curl command:
curl "https://jdx35ariibmj1auz.us-east-1.aws.endpoints.huggingface.cloud/v1/chat/completions" \
-X POST \
-H "Authorization: Bearer TOKEN" \
-H "Content-Type: application/json" \
-d '{
"model": "tgi",
"messages": [
{
"role": "user",
"content": "What is deep learning?"
}
],
"max_tokens": 150,
"stream": false
}'
{"object":"chat.completion","id":"","created":1729893704,"model":"/repository","system_fingerprint":"2.3.1-sha-a094729","choices":[{"index":0,"message":{"role":"assistant","content":"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"},"logprobs":null,"finish_reason":"length"}],"usage":{"prompt_tokens":205,"completion_tokens":150,"total_tokens":355}}
I'm running on the: GPU · Nvidia T4
It also seems EXTREMELY slow.
Ditto
Yeah, even using the apply template still gives me all !!!!!!!!!!!!!!!!!
<|begin_of_text|><|start_header_id|>user<|end_header_id|>
Task: Check if there is unsafe content in 'User' messages in conversations according our safety policy with the below categories.
<BEGIN UNSAFE CONTENT CATEGORIES>
S1: Violent Crimes.
S2: Non-Violent Crimes.
S3: Sex Crimes.
S4: Child Exploitation.
S5: Defamation.
S6: Specialized Advice.
S7: Privacy.
S8: Intellectual Property.
S9: Indiscriminate Weapons.
S10: Hate.
S11: Self-Harm.
S12: Sexual Content.
S13: Elections.
S14: Code Interpreter Abuse.
<END UNSAFE CONTENT CATEGORIES>
<BEGIN CONVERSATION>
User: I love everyone
<END CONVERSATION>
Provide your safety assessment for ONLY THE LAST User message in the above conversation:
- First line must read'safe' or 'unsafe'.
- If unsafe, a second line must include a comma-separated list of violated categories.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Would be nice if whoever made this project would care about it enough to fix it......