File size: 5,467 Bytes
ce6ec82 1681022 ce6ec82 1681022 ce6ec82 e08207b ce6ec82 1681022 ce6ec82 1681022 e08207b 0ae8221 e08207b 0ae8221 1681022 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 |
---
base_model: unsloth/Phi-3.5-mini-instruct
language:
- en
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
---
# This page is work in progress!
## Overview
The **Fhi-3.5-mini-instruct** is a fine-tuned version of the [unsloth/Phi-3.5-mini-instruct](https://huggingface.co/unsloth/Phi-3.5-mini-instruct) model, optimized for function-calling.
### Usage
Here’s a basic example of how to use function calling with the Fhi-3.5-mini-instruct model:
```python
def get_current_temperature(location: str) -> float:
"""
Get the current temperature at a location.
Args:
location: The location to get the temperature for, in the format "City, Country"
Returns:
The current temperature at the specified location in the specified units, as a float.
"""
return 22.
# Create the messages list
messages = [
{"role": "system", "content": "You are a helpful weather assistant."},
{"role": "user", "content": "What's the current weather in London and New York? Please use Celsius."}
]
# Apply the chat template
prompt = tokenizer.apply_chat_template(
messages,
tools=[get_current_temperature], # Pass the custom tool
add_generation_prompt=True,
tokenize=False
)
inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False, num_return_sequences=1, use_cache=True, temperature=0.001, top_p=1, eos_token_id=[32007])
resu = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True)
print(resu)
```
The result will look like this:
```python
[
{'name': 'get_current_temperature', 'arguments': {'location': 'London, UK'}},
{'name': 'get_current_temperature', 'arguments': {'location': 'New York, USA'}}
]
```
## Testing and Benchmarking
This model is still undergoing testing and evaluation. Use it at your own risk until further validation is complete. Performance on benchmarks like MMLU and MMLU-Pro will be updated soon.
| Benchmark | Fhi-3.5 Mini-Ins | Phi-3.5 Mini-Ins | Mistral-7B-Instruct-v0.3 | Mistral-Nemo-12B-Ins-2407 | Llama-3.1-8B-Ins | Gemma-2-9B-Ins | Gemini 1.5 Flash | GPT-4o-mini-2024-07-18 (Chat) |
|----------------------------|------------------|------------------|--------------------------|---------------------------|------------------|----------------|------------------|-------------------------------|
| Multilingual MMLU | ____ | 55.4 | 47.4 | 58.9 | 56.2 | 63.8 | 77.2 | 72.9 |
| MMLU (5-shot) | __ | 69 | 60.3 | 67.2 | 68.1 | 71.3 | 78.7 | 77.2 |
| MMLU-Pro (3-shot, CoT) |__| 47.4 | 18 | 40.7 | 44 | 50.1 | 57.2 | 62.8 |
<!-- | Multilingual MMLU-Pro | 30.9 | 30.21 | 15.0 | 34.0 | 21.4 | 43.0 | 57.9 | 53.2 |
| **Average** | **55.2** | **52.3** | **47.9** | **55.3** | **47.5** | **59.6** | **64.3** | **76.6** |
-->
<!-- The table below shows Multilingual MMLU scores in some of the supported languages.
| Benchmark | Phi-3.5 Mini-Ins | Phi-3.0-Mini-128k-Instruct (June2024) | Mistral-7B-Instruct-v0.3 | Mistral-Nemo-12B-Ins-2407 | Llama-3.1-8B-Ins | Gemma-2-9B-Ins | Gemini 1.5 Flash | GPT-4o-mini-2024-07-18 (Chat) |
|-----------|------------------|-----------------------|--------------------------|---------------------------|------------------|----------------|------------------|-------------------------------|
| Arabic | 44.2 | 35.4 | 33.7 | 45.3 | 49.1 | 56.3 | 73.6 | 67.1 |
| Chinese | 52.6 | 46.9 | 45.9 | 58.2 | 54.4 | 62.7 | 66.7 | 70.8 |
| Dutch | 57.7 | 48.0 | 51.3 | 60.1 | 55.9 | 66.7 | 80.6 | 74.2 |
| French | 61.1 | 61.7 | 53.0 | 63.8 | 62.8 | 67.0 | 82.9 | 75.6 |
| German | 62.4 | 61.3 | 50.1 | 64.5 | 59.9 | 65.7 | 79.5 | 74.3 |
| Italian | 62.8 | 63.1 | 52.5 | 64.1 | 55.9 | 65.7 | 82.6 | 75.9 |
| Russian | 50.4 | 45.3 | 48.9 | 59.0 | 57.4 | 63.2 | 78.7 | 72.6 |
| Spanish | 62.6 | 61.3 | 53.9 | 64.3 | 62.6 | 66.0 | 80.0 | 75.5 |
| Ukrainian | 45.2 | 36.7 | 46.9 | 56.6 | 52.9 | 62.0 | 77.4 | 72.6 |
-->
## Credits
Will be updated soon |