Upload tokenizer_config.json

README.md

@@ -1,307 +1,3 @@
----
-language:
-- en
-- fr
-- es
-- pt
-tags:
-- falcon3
-base_model: tiiuae/Falcon3-7B-Base
-license: other
-license_name: falcon-llm-license
-license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
-library_name: transformers
----
-
-<div align="center">
-    <img src="https://huggingface.co/datasets/tiiuae/documentation-images/resolve/main/general/falco3-logo.png" alt="drawing" width="500"/>
-</div>
-
-# Falcon3-7B-Instruct
-
-**Falcon3** family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B.
-
-This repository contains the **Falcon3-7B-Instruct**. It achieves state of art results (at the time of release) on reasoning, language understanding, instruction following, code and mathematics tasks.
-Falcon3-7B-Instruct supports 4 languages (english, french, spanish, portuguese) and a context length up to 32K.
-
-## Model Details
-- Architecture
-  - Transformer based causal decoder only architecture
-  - 28 decoder blocks
-  - Grouped query attention (GQA) for faster inference: 12 query heads and 4 key value heads
-  - Wider head dimension: 256
-  - High RoPE value to support long context understanding: 1000042
-  - Uses SwiGLU and RMSNorm
-  - 32K context length
-  - 131K vocab size
-- Pretrained on 14 Teratokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 1024 H100 GPU chips
-- Postrained on 1.2 million samples of STEM, conversations, code, safety and function call data
-- Supports EN, FR, ES, PT
-- Developed by [Technology Innovation Institute](https://www.tii.ae)
-- License: TII Falcon-LLM License 2.0
-- Model Release Date: December 2024
-
-
-## Getting started
-
-<details>
-<summary> Click to expand </summary>
-
-```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
-
-
-from transformers import AutoModelForCausalLM, AutoTokenizer
-
-model_name = "tiiuae/Falcon3-7B-Instruct"
-
-model = AutoModelForCausalLM.from_pretrained(
-    model_name,
-    torch_dtype="auto",
-    device_map="auto"]
-)
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-
-prompt = "How many hours in one day?"
-messages = [
-    {"role": "system", "content": "You are a helpful friendly assistant Falcon3 from TII, try to follow instructions as much as possible."},
-    {"role": "user", "content": prompt}
-]
-text = tokenizer.apply_chat_template(
-    messages,
-    tokenize=False,
-    add_generation_prompt=True
-)
-model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
-
-generated_ids = model.generate(
-    **model_inputs,
-    max_new_tokens=1024
-)
-generated_ids = [
-    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
-]
-
-response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
-print(response)
-```
-
-</details>
-
-<br>
-
-## Benchmarks
-We report the official HuggingFace leaderboard normalized evaluations [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) in the following table.
-<table border="1" style="width: 100%; text-align: center; border-collapse: collapse;">
-    <colgroup>
-        <col style="width: 10%;">
-        <col style="width: 7%;">
-        <col style="width: 7%;">
-        <col style="background-color: rgba(80, 15, 213, 0.5); width: 7%;">
-    </colgroup>
-    <thead>
-        <tr>
-            <th>Benchmark</th>
-            <th>Llama-3.1-8B-Instruct</th>
-            <th>Qwen2.5-7B-Instruct</th>
-            <th>Falcon3-7B-Instruct</th>
-        </tr>
-    </thead>
-    <tbody>
-        <tr>
-            <td>IFEval</td>
-            <td><b>78.56</b></td>
-            <td>75.85</td>
-            <td>76.12</td>
-        </tr>
-        <tr>
-            <td>BBH (3-shot)</td>
-            <td>29.89</td>
-            <td>34.89</td>
-            <td><b>37.92</b></td>
-        </tr>
-        <tr>
-            <td>MATH Lvl-5 (4-shot)</td>
-            <td>19.34</td>
-            <td>0.00</td>
-            <td><b>31.87</b></td>
-        </tr>              
-        <tr>
-            <td>GPQA (0-shot)</td>
-            <td>2.35</td>
-            <td>5.48</td>
-            <td><b>8.05</b></td>
-        </tr>
-        <tr>
-            <td>MUSR (0-shot)</td>
-            <td>8.41</td>
-            <td>8.45</td>
-            <td><b>21.17</b></td>
-        </tr>
-        <tr>
-            <td>MMLU-PRO (5-shot)</td>
-            <td>30.68</td>
-            <td><b>36.52</b></td>
-            <td>34.30</td>
-        </tr>        
-    </tbody>
-</table>
-
-Also, we report in the following table our internal pipeline benchmarks.
- - We use [lm-evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness).
- - We report **raw scores** obtained by applying chat template and fewshot_as_multiturn.
- - We use same batch-size across all models.
-
-<table border="1" style="width: 100%; text-align: center; border-collapse: collapse;">
-    <colgroup>
-        <col style="width: 10%;">
-        <col style="width: 10%;">
-        <col style="width: 7%;">
-        <col style="width: 7%;">
-        <col style="background-color: rgba(80, 15, 213, 0.5); width: 7%;">
-    </colgroup>
-    <thead>
-        <tr>
-            <th>Category</th>
-            <th>Benchmark</th>
-            <th>Llama-3.1-8B-Instruct</th>
-            <th>Qwen2.5-7B-Instruct</th>
-            <th>Falcon3-7B-Instruct</th>
-        </tr>
-    </thead>
-    <tbody>
-        <tr>
-            <td rowspan="3">General</td>
-            <td>MMLU (5-shot)</td>
-            <td>68.2</td>
-            <td><b>73.5</b></td>
-            <td>70.5</td>
-        </tr>
-        <tr>
-            <td>MMLU-PRO (5-shot)</td>
-            <td>36.4</td>
-            <td><b>43.1</b></td>
-            <td>40.7</td>
-        </tr>
-        <tr>
-            <td>IFEval</td>
-            <td><b>78.8</b></td>
-            <td>74.7</td>
-            <td>76.5</td>
-        </tr>
-        <tr>
-            <td rowspan="3">Math</td>
-            <td>GSM8K (5-shot)</td>
-            <td><b>82.6</b></td>
-            <td>72.0</td>
-            <td>81.4</td>
-        </tr>
-        <tr>
-            <td>GSM8K (8-shot, COT)</td>
-            <td><b>85.4</b></td>
-            <td>76.6</td>
-            <td>79.7</td>
-        </tr>
-        <tr>
-            <td>MATH Lvl-5 (4-shot)</td>
-            <td>15.4</td>
-            <td>-</td>
-            <td><b>29.4</b></td>
-        </tr>
-        <tr>
-            <td rowspan="5">Reasoning</td>
-            <td>Arc Challenge (25-shot)</td>
-            <td>58.6</td>
-            <td>57.8</td>
-            <td><b>62.6</b></td>
-        </tr>
-        <tr>
-            <td>GPQA (0-shot)</td>
-            <td><b>33.5</b></td>
-            <td>32</td>
-            <td>31.9</td>
-        </tr>
-        <tr>
-            <td>GPQA (0-shot, COT)</td>
-            <td>9.6</td>
-            <td>13.8</td>
-            <td><b>22.3</b></td>
-        </tr>
-        <tr>
-            <td>MUSR (0-shot)</td>
-            <td>38.6</td>
-            <td>41</td>
-            <td><b>46.4</b></td>
-        </tr>
-        <tr>
-            <td>BBH (3-shot)</td>
-            <td>48.6</td>
-            <td><b>54.1</b></td>
-            <td>52.4</td>
-        </tr>
-        <tr>
-            <td rowspan="4">CommonSense Understanding</td>
-            <td>PIQA (0-shot)</td>
-            <td><b>78.9</b></td>
-            <td>73.7</td>
-            <td>78.8</td>
-        </tr>
-        <tr>
-            <td>SciQ (0-shot)</td>
-            <td>80.2</td>
-            <td>50.9</td>
-            <td><b>94.7</b></td>
-        </tr>
-        <tr>
-            <td>Winogrande (0-shot)</td>
-            <td>-</td>
-            <td>-</td>
-            <td>70.4</td>
-        </tr>
-        <tr>
-            <td>OpenbookQA (0-shot)</td>
-            <td><b>46.2</b></td>
-            <td>42.4</td>
-            <td>45.8</td>
-        </tr>
-        <tr>
-            <td rowspan="2">Instructions following</td>
-            <td>MT-Bench (avg)</td>
-            <td>7.9</td>
-            <td><b>8.5</b></td>
-            <td>8.4</td>
-        </tr>
-        <tr>
-            <td>Alpaca (WC)</td>
-            <td>26.6</td>
-            <td><b>31.5</b></td>
-            <td>26.1</td>
-        </tr>
-        <tr>
-            <td>Tool use</td>
-            <td>BFCL AST (avg)</td>
-            <td>90.6</td>
-            <td><b>91.4</b></td>
-            <td>89.5</td>
-        </tr>
-    </tbody>
-</table>
-
-## Useful links
-- View our [release blogpost](https://huggingface.co/blog/falcon3).
-- Feel free to join [our discord server](https://discord.gg/fwXpMyGc) if you have any questions or to interact with our researchers and developers.
-
-## Technical Report
-Coming soon....
-
-## Citation
-If Falcon3 family were helpful to your work, feel free to give us a cite.
-
-```
-@misc{Falcon3,
-    title = {The Falcon 3 family of Open Models},
-    author = {TII Team},
-    month = {December},
-    year = {2024}
-}
-```
+---
+license: apache-2.0
+---

config.json

@@ -1,4 +1,5 @@
 {
+  "_name_or_path": "Iheb-Chaabane/falcon3-7b-explore-dpo-bs-64",
   "architectures": [
     "LlamaForCausalLM"
   ],
@@ -9,6 +10,7 @@
   "head_dim": 256,
   "hidden_act": "silu",
   "hidden_size": 3072,
+  "initializer_range": 0.02,
   "intermediate_size": 23040,
   "max_position_embeddings": 32768,
   "mlp_bias": false,
@@ -24,5 +26,5 @@
   "torch_dtype": "bfloat16",
   "transformers_version": "4.46.1",
   "use_cache": true,
-  "vocab_size": 131072
+  "vocab_size": 131080
 }

model-00001-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:07e279c8c6075600e5dc795364efff8897de0f0c22a1d2d8db79a70adf8edb3f
-size 4938900432
+oid sha256:ada95243d59a5b8a5d60ea7bec7907ca20b92bb124d5054b51792fe059b72195
+size 4938949584

model-00002-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c5d6600f34e9972eed3201425ba75c2d58f574655f373ea8b86ddfa37d391f2a
+oid sha256:c0f167f4dc5fb028251a03f67ce36bef07a163084fbd8f7d63ca043d770ab9ca
 size 4942085160

model-00003-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a96480584a0b5bd09c556e53d952146008bb423e5e12ea9bbd0b60d62f9a2f72
+oid sha256:11a6edf04d6b4ab1044d88107eb8a4c71d6378c7d232c3c668870ceae1d2a80c
 size 4224838512

model-00004-of-00004.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0b84ea911989e21ebf4ac05018171f73016d8ae72b7904e89289be0b4672a403
-size 805306496
+oid sha256:4be476e54f6ce54be4690cb9b7241959fd2096ab9a4b97648679e1fce43c575b
+size 805355648

model.safetensors.index.json

@@ -1,6 +1,6 @@
 {
   "metadata": {
-    "total_size": 14911113216
+    "total_size": 14911199232
   },
   "weight_map": {
     "lm_head.weight": "model-00004-of-00004.safetensors",

special_tokens_map.json

@@ -32,7 +32,7 @@
     "single_word": false
   },
   "pad_token": {
-    "content": "<|pad|>",
+    "content": "<pad>",
     "lstrip": false,
     "normalized": false,
     "rstrip": false,

tokenizer.json

@@ -18212,7 +18212,16 @@
     },
     {
       "id": 2023,
-      "content": "<|pad|>",
+      "content": ">>UNUSED_1897<<",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 131072,
+      "content": "<pad>",
       "single_word": false,
       "lstrip": false,
       "rstrip": false,
@@ -20280,7 +20289,7 @@
       ">>UNUSED_1894<<": 2020,
       ">>UNUSED_1895<<": 2021,
       ">>UNUSED_1896<<": 2022,
-      "<|pad|>": 2023,
+      ">>UNUSED_1897<<": 2023,
       "!": 2024,
       "\"": 2025,
       "#": 2026,

tokenizer_config.json

@@ -16186,7 +16186,15 @@
       "special": true
     },
     "2023": {
-      "content": "<|pad|>",
+      "content": ">>UNUSED_1897<<",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "131072": {
+      "content": "<pad>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -16219,15 +16227,11 @@
     ">>PASSWORD<<",
     ">>KEY<<"
   ],
-  "chat_template": "{%- if tools %}\n{{- '<|system|>\\n' }}\n{%- if messages[0]['role'] == 'system' %}\n{{- messages[0]['content'] }}\n{%- set remaining_messages = messages[1:] %}\n{%- else %}\n{%- set remaining_messages = messages %}\n{%- endif %}\n{{- 'You are a Falcon assistant skilled in function calling. You are helpful, respectful, and concise.\\n\\n# Tools\\n\\nYou have access to the following functions. You MUST use them to answer questions when needed. For each function call, you MUST return a JSON object inside <tool_call></tool_call> tags.\\n\\n<tools>' + tools|tojson(indent=2) + '</tools>\\n\\n# Output Format\\n\\nYour response MUST follow this format when making function calls:\\n<tool_call>\\n[\\n  {\"name\": \"function_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}},\\n  {\"name\": \"another_function\", \"arguments\": {\"arg\": \"value\"}}\\n]\\n</tool_call>\\nIf no function calls are needed, respond normally without the tool_call tags.\\n' }}\n{%- for message in remaining_messages %}\n{%- if message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if message.content %}\n{{- '<|assistant|>\\n' + message['content'] }}\n{%- endif %}\n{%- if message.tool_calls %}\n{{- '\\n<tool_call>\\n' }}\n{{- message.tool_calls|tojson(indent=2) }}\n{{- '\\n</tool_call>' }}\n{%- endif %}\n{{- eos_token + '\\n' }}\n{%- elif message['role'] == 'tool' %}\n{{- '<|assistant|>\\n<tool_response>\\n' + message['content'] + '\\n</tool_response>\\n' }}\n{%- endif %}\n{%- endfor %}\n{{- '<|assistant|>\\n' if add_generation_prompt }}\n{%- else %}\n{%- for message in messages %}\n{%- if message['role'] == 'system' %}\n{{- '<|system|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if not loop.last %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token + '\\n' }}\n{%- else %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token }}\n{%- endif %}\n{%- endif %}\n{%- if loop.last and add_generation_prompt %}\n{{- '<|assistant|>\\n' }}\n{%- endif %}\n{%- endfor %}\n{%- endif %}",
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'system' %}{{ '<|system|>\n' + message['content'] + '\n' }}{% elif message['role'] == 'user' %}{{ '<|user|>\n' + message['content'] + '\n' }}{% elif message['role'] == 'assistant' %}{% if not loop.last %}{{ '<|assistant|>\n'  + message['content'] + eos_token + '\n' }}{% else %}{{ '<|assistant|>\n'  + message['content'] + eos_token }}{% endif %}{% endif %}{% if loop.last and add_generation_prompt %}{{ '<|assistant|>\n' }}{% endif %}{% endfor %}",
   "clean_up_tokenization_spaces": true,
   "eos_token": "<|endoftext|>",
-  "extra_special_tokens": {},
-  "model_input_names": [
-    "input_ids",
-    "attention_mask"
-  ],
   "model_max_length": 32768,
-  "pad_token": "<|pad|>",
+  "pad_token": "<pad>",
   "tokenizer_class": "PreTrainedTokenizerFast"
 }
+