mistralai

/

Mistral-Nemo-Instruct-2407

like 1.34k

Follow

Mistral AI_ 3.72k

Is there something wrong with the way I use it? Why is the system prompt missing? How can I use the tokenizer of this model?

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-Nemo-Instruct-2407")
msg = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's the weather like today?"},
    {"role": "assistant", "content": "The weather is sunny today."},
    {"role": "user", "content": "Can you recommend a good restaurant?"},
    {"role": "tool_results", "content": {"content": "Here are some recommendations: ..."}, "tool_call_id": "abcdefghi"}
]

output = tokenizer.apply_chat_template(msg,tokenize=False,add_generation_prompt=True,tools=tools,)
print(output)

the output is : 

<s>[INST]What's the weather like today?[/INST]The weather is sunny today.</s>[AVAILABLE_TOOLS][{"type": "function", "function": {"name": "get_current_weather", "description": "Get the current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "format": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location."}}, "required": ["location", "format"]}}}][/AVAILABLE_TOOLS][INST]Can you recommend a good restaurant?[/INST][TOOL_RESULTS]{"content": Here are some recommendations: ..., "call_id": "abcdefghi"}[/TOOL_RESULTS]

I changed the calling method below, but the system prompt is always before the last user instead of at the beginning.

from mistral_inference.transformer import Transformer
from mistral_inference.generate import generate

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.protocol.instruct.messages import UserMessage,SystemMessage,AssistantMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest
mistral_models_path = "mistralai/Mistral-Nemo-Instruct-2407"
tokenizer = MistralTokenizer.from_file(f"{mistral_models_path}/tekken.json")
# model = Transformer.from_folder(mistral_models_path)

user = "user1"
system_prompt ="You are helpfull assistant,help humman as much as you can"
completion_request = ChatCompletionRequest(messages=[SystemMessage(content=system_prompt),
                                                    
                                                     UserMessage(content=user),
                                                     AssistantMessage(content='assis1'),
                                                     UserMessage(content="user2"),
                                                     AssistantMessage(content='assis2'),
                                                     UserMessage(content="user3"),
                                                    ])

tokens = tokenizer.encode_chat_completion(completion_request).text

print(tokens)
#<s>[INST]user1[/INST]assis1</s>[INST]user2[/INST]assis2</s>[INST]You are helpfull assistant,help humman as much as you can\n\nuser3[/INST]

how to use it to output format like 
<s><SYS>You are helpfull assistant,help humman as much as you can\n\n</SYS>[INST]user1[/INST]assis1</s>[INST]user2[/INST]assis2</s>[INST]user3[/INST]