File size: 15,678 Bytes
b597d38 befff52 b721e32 b3aed60 b721e32 1920b0e 70840e3 1920b0e b597d38 c1bced4 1920b0e c1bced4 568e186 c1bced4 f973676 c1bced4 3e9b5e2 c1bced4 125cb7c c1bced4 3e9b5e2 c1bced4 3e9b5e2 c1bced4 babaf3f c1bced4 3e9b5e2 c1bced4 568e186 125cb7c 3e9b5e2 125cb7c 568e186 c1bced4 568e186 c1bced4 125cb7c c1bced4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 |
---
license: mit
library_name: transformers
pipeline_tag: text-generation
datasets:
- yulan-team/YuLan-Mini-Datasets
- HuggingFaceFW/fineweb-edu
- bigcode/the-stack-v2
- mlfoundations/dclm-baseline-1.0
- math-ai/AutoMathText
- gair-prox/open-web-math-pro
- RUC-AIBOX/long_form_thought_data_5k
- internlm/Lean-Workbook
- internlm/Lean-Github
- deepseek-ai/DeepSeek-Prover-V1
- ScalableMath/Lean-STaR-base
- ScalableMath/Lean-STaR-plus
- ScalableMath/Lean-CoT-base
- ScalableMath/Lean-CoT-plus
- opencsg/chinese-fineweb-edu
- liwu/MNBVC
- vikp/textbook_quality_programming
- HuggingFaceTB/smollm-corpus
- OpenCoder-LLM/opc-annealing-corpus
- OpenCoder-LLM/opc-sft-stage1
- OpenCoder-LLM/opc-sft-stage2
- XinyaoHu/AMPS_mathematica
- deepmind/math_dataset
- mrfakename/basic-math-10m
- microsoft/orca-math-word-problems-200k
- AI-MO/NuminaMath-CoT
- HuggingFaceTB/cosmopedia
- MU-NLPC/Calc-ape210k
- manu/project_gutenberg
- storytracer/LoC-PD-Books
- allenai/dolma
language:
- en
- zh
tags:
- code
- math
arxiv: 2412.17743
model-index:
- name: YuLan-Mini
results:
- task:
type: text-generation
dataset:
type: openai_humaneval
name: HumanEval
metrics:
- name: pass@1
type: pass@1
value: 0.640
verified: false
- task:
type: text-generation
dataset:
type: mbpp
name: MBPP
metrics:
- name: pass@1
type: pass@1
value: 0.659
verified: false
- task:
type: text-generation
dataset:
type: math-500
name: MATH-500
metrics:
- name: maj@1
type: maj@1
value: 0.378
verified: false
- task:
type: text-generation
dataset:
type: gsm8k
name: GSM8K
metrics:
- name: maj@1
type: maj@1
value: 0.684
verified: false
---
# Important Notice: This is a pre-trained **base model** without instruction-following capabilities. The **SFT version** will be released within a few weeks.
<div align=center>
<img src="assets/YuLan-logo.jpg" width="400px">
<h1>YuLan-Mini: An Open Data-efficient Language Model</h1>
<a href="https://github.com/RUC-GSAI/YuLan-Mini/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue" alt="license"></a>
<a href="https://arxiv.org/abs/2412.17743" target="_blank"><img src=https://img.shields.io/badge/arXiv-b5212f.svg?logo=arxiv></a>
<a href="https://huggingface.co/collections/yulan-team/yulan-mini-676d214b24376739b00d95f3"><img alt="Static Badge" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-blue?color=8A2BE2"></a>
<a href="https://github.com/RUC-GSAI/YuLan-Mini" target="_blank"><img src="https://img.shields.io/github/stars/RUC-GSAI/YuLan-Mini"></a>
</div>
YuLan-Mini is a lightweight language model with 2.4 billion parameters. It achieves performance comparable to industry-leading models trained on significantly more data, despite being pre-trained on only 1.08T tokens. The model excels particularly in the domains of **mathematics** and **code**. To facilitate reproducibility, we will open-source the relevant [pre-training resources](https://github.com/RUC-GSAI/YuLan-Mini/tree/main/pretrain).
---
## Model Downloads π
| Model | Context Length | SFT |
|---------|----------------|-----|
| [YuLan-Mini](https://huggingface.co/yulan-team/YuLan-Mini) (Recommended) | 28K | β |
| [YuLan-Mini-2.4B-4K](https://huggingface.co/yulan-team/YuLan-Mini-Intermediate-4K) | 4K | β |
| YuLan-Mini-Instruct | Comming soon | β
|
---
## Features π
<div align=center>
<img src="assets/main.png">
</div>
Our pre-training methodology improves training efficiency through three key innovations:
1. an elaborately designed **data pipeline** that combines data cleaning with data schedule strategies;
2. a systematic **optimization method** that can effectively mitigate training instability;
3. an effective **annealing approach** that integrate targeted data selection and long context training.
---
## Behchmarks π
| Models | Model Size | # Train Tokens | Context Length | MATH 500 | GSM 8K | Human Eval | MBPP | RACE Middle | RACE High | RULER |
|:----------------|----------:|--------------:|--------------:|:--------|:------|:----------|:------|:-----------|:---------|:------|
| MiniCPM | 2.6B | 1.06T | 4K | 15.00 | 53.83 | 50.00* | 47.31 | 56.61 | 44.27 | N/A |
| Qwen-2 | 1.5B | 7T | 128K | 22.60 | 46.90* | 34.80* | 46.90* | 55.77 | 43.69 | 60.16 |
| Qwen2.5 | 0.5B | 18T | 128K | 23.60 | 41.60* | 30.50* | 39.30* | 52.36 | 40.31 | 49.23 |
| Qwen2.5 | 1.5B | 18T | 128K | **45.40** | **68.50\*** | 37.20* | 60.20* | **58.77** | 44.33 | <ins>68.26</ins> |
| Gemma2 | 2.6B | 2T | 8K | 18.30* | 30.30* | 19.50* | 42.10* | - | - | N/A |
| StableLM2 | 1.7B | 2T | 4K | - | 20.62 | 8.50* | 17.50 | 56.33 | **45.06** | N/A |
| SmolLM2 | 1.7B | 11T | 8K | 11.80 | - | 23.35 | 45.00 | 55.77 | 43.06 | N/A |
| Llama3.2 | 3.2B | 9T | 128K | 7.40 | - | 29.30 | 49.70 | 55.29 | 43.34 | **77.06** |
| YuLan-Mini | 2.4B | 1.04T | 4K | 32.60 | 66.65 | <ins>61.60</ins> | **66.70** | 55.71 | 43.58 | N/A |
| YuLan-Mini | 2.4B | 1.08T | 28K | <ins>37.80</ins> | <ins>68.46</ins> | **64.00** | <ins>65.90</ins>| <ins>57.18</ins> | <ins>44.57</ins> | 51.48 |
| Models | LAMBADA | MMLU | CMMLU | CEval | HellaSwag | WinoGrande | StoryCloze | ARC-e | ARC-c |
|:----------------|:-------|:-----|:-----|:-----|:----------|:-----------|:-----------|:-----|:-----|
| MiniCPM-2.6B | 61.91 | 53.37 | 48.97 | 48.24 | 67.92 | 65.74 | 78.51 | 55.51 | 43.86 |
| Qwen2-1.5B | 64.68 | 55.90 | **70.76** | **71.94** | 66.11 | 66.14 | 77.60 | 62.21 | 42.92 |
| Qwen2.5-0.5B | 52.00 | 47.50 | 52.17 | 54.27 | 50.54 | 55.88 | 71.67 | 56.10 | 39.51 |
| Qwen2.5-1.5B | 62.12 | <ins>60.71</ins> | <ins>67.82</ins> | <ins>69.05</ins> | 67.18 | 64.48 | 76.80 | **71.51** | <ins>53.41</ins> |
| Gemma2-2.6B | - | 52.20*| - | 28.00*| <ins>74.60*</ins> | **71.50\*** | - | - | **55.70\***|
| StableLM2-1.7B | 66.15 | 40.37 | 29.29 | 26.99 | 69.79 | 64.64 | <ins>78.56</ins> | 54.00 | 40.78 |
| SmolLM2-1.7B | <ins>67.42</ins> | 51.91 | 33.46 | 35.10 | 72.96 | 67.40 | **79.32** | 44.82 | 35.49 |
| Llama3.2-3B | **69.08** | **63.40** | 44.44 | 44.49 | **75.62** | <ins>67.48</ins> | 76.80 | <ins>70.12</ins> | 48.81 |
| YuLan-Mini | 64.72 | 51.79 | 48.35 | 51.47 | 68.65 | 67.09 | 76.37 | 69.87 | 50.51 |
| YuLan-Mini | 65.67 | 49.10 | 45.45 | 48.23 | 67.22 | 67.24 | 75.89 | 67.47 | 49.32 |
---
## Pre-Training Resources π§
To enhance research transparency and reproducibility, we are open-sourcing relevant [pre-training resources](https://github.com/RUC-GSAI/YuLan-Mini/blob/main/pretrain):
<details><summary>1. Pre-training and Evaluation Code</summary>
The pre-training and evaluation code will be released in a future update.
</details>
<details><summary>2. Intermediate Stage Checkpoints</summary>
The intermediate stage checkpoints are released in <a href="https://huggingface.co/collections/yulan-team/yulan-mini-676d214b24376739b00d95f3">YuLan-Mini</a>.
<table>
<thead>
<tr>
<th>Stage</th>
<th>Curriculum Phase</th>
<th>4K Context</th>
<th>28K Context</th>
<th>Optimizer</th>
<th>Inference Architecture</th>
<th>LAMBADA <code>Acc</code></th>
<th>GSM8K <code>Acc</code></th>
<th>HumanEval <code>pass@1</code></th>
</tr>
</thead>
<tbody>
<tr>
<td>Stable</td>
<td>5</td>
<td><a href="https://huggingface.co/yulan-team/YuLan-Mini-Phase5">YuLan-Mini-Phase5</a></td>
<td></td>
<td></td>
<td><code>yulanmini</code></td>
<td>53.85</td>
<td>3.41</td>
<td>12.26</td>
</tr>
<tr>
<td>Stable</td>
<td>10</td>
<td><a href="https://huggingface.co/yulan-team/YuLan-Mini-Phase10">YuLan-Mini-Phase10</a></td>
<td></td>
<td></td>
<td><code>yulanmini</code></td>
<td>55.00</td>
<td>9.57</td>
<td>15.95</td>
</tr>
<tr>
<td>Stable</td>
<td>15</td>
<td><a href="https://huggingface.co/yulan-team/YuLan-Mini-Phase15">YuLan-Mini-Phase15</a></td>
<td></td>
<td></td>
<td><code>yulanmini</code></td>
<td>55.81</td>
<td>13.81</td>
<td>16.99</td>
</tr>
<tr>
<td>Stable</td>
<td>20</td>
<td><a href="https://huggingface.co/yulan-team/YuLan-Mini-Phase20">YuLan-Mini-Phase20</a></td>
<td></td>
<td>β
</td>
<td><code>yulanmini</code></td>
<td>55.81</td>
<td>21.39</td>
<td>20.79</td>
</tr>
<tr>
<td>Stable</td>
<td>25 (1T tokens)</td>
<td><a href="https://huggingface.co/yulan-team/YuLan-Mini-Before-Annealing">YuLan-Mini-Before-Annealing</a></td>
<td></td>
<td>β
</td>
<td><code>yulanmini</code></td>
<td>55.67</td>
<td>29.94</td>
<td>34.06</td>
</tr>
<tr>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>Annealing</td>
<td>26</td>
<td>YuLan-Mini-4K</td>
<td></td>
<td></td>
<td><code>llama</code>*</td>
<td>64.72</td>
<td>66.65</td>
<td>61.60</td>
</tr>
<tr>
<td>Annealing</td>
<td>27</td>
<td></td>
<td><a href="https://huggingface.co/yulan-team/YuLan-Mini">YuLan-Mini</a></td>
<td></td>
<td><code>llama</code>*</td>
<td>65.67</td>
<td>68.46</td>
<td>64.00</td>
</tr>
</tbody>
</table>
\*: For easier inference and deployment, we merged the re-parameterized added parameters and scaling factors into the final released models ([**YuLan-Mini**](https://huggingface.co/yulan-team/YuLan-Mini) and **YuLan-Mini-Intermediate-4K**), enabling it to run on the Llama architecture. However, these parameters are still retained in the intermediate checkpoints from the training process.
</details>
<details><summary>3. Optimizer States Before Annealing</summary>
<a href="https://huggingface.co/yulan-team/YuLan-Mini-Before-Annealing">YuLan-Mini-Before-Annealing</a>
</details>
<details><summary>4. The Used Open-Source Datasets </summary>
<a href="https://github.com/RUC-GSAI/YuLan-Mini/blob/main/pretrain/datasets">Used-Datasets-List</a>
</details>
<details><summary>5. Data Distribution for every phase</summary>
<a href="https://github.com/RUC-GSAI/YuLan-Mini/blob/main/pretrain/datasets/final.pdf">
<div align=center>
<img src="assets/data_distribution_for_every_phase.png">
</div>
</a>
</details>
<details><summary>6. Synthetic Data</summary>
Data cleaning and synthesis pipeline:
<div align=center>
<img src="https://github.com/RUC-GSAI/YuLan-Mini/blob/main/assets/data-pipeline.png">
</div>
The synthetic data we are using is released in <a href="https://huggingface.co/collections/yulan-team/yulan-mini-676d214b24376739b00d95f3">YuLan-Mini-Datasets</a>
</details>
### What you can do with these pre-training resources
1. **Pre-train** your own LLM. You can use [our data](https://huggingface.co/yulan-team/YuLan-Mini-Datasets) and curriculum to train a model that's just as powerful as YuLan-Mini.
2. Perform your own **learning rate annealing**. During the annealing phase, YuLan-Mini's learning ability is at its peak. You can resume training from [the checkpoint before annealing](https://huggingface.co/yulan-team/YuLan-Mini-Before-Annealing) and use your own dataset for learning rate annealing.
3. **Fine-tune** the Instruct version of the LLM. You can use the [YuLan-Mini](https://huggingface.co/yulan-team/YuLan-Mini) base model to train your own Instruct version.
4. **Training dynamics** research. You can use YuLan-Mini's [intermediate checkpoints](https://huggingface.co/collections/yulan-team/yulan-mini-676d214b24376739b00d95f3) to explore internal changes during the pre-training process.
5. **Synthesize** your own data. You can use YuLan-Mini's [data pipeline](https://github.com/RUC-GSAI/YuLan-Mini) to clean and generate your own dataset.
---
## Quick Start π»
Below is a simple example for inference using Huggingface:
**Huggingface Inference Example**
```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("yulan-team/YuLan-Mini")
model = AutoModelForCausalLM.from_pretrained("yulan-team/YuLan-Mini", torch_dtype=torch.bfloat16)
# Input text
input_text = "Renmin University of China is"
inputs = tokenizer(input_text, return_tensors="pt")
# Completion
output = model.generate(inputs["input_ids"], max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
**vLLM Serve Example**
```bash
vllm serve yulan-team/YuLan-Mini --dtype bfloat16
```
**SGLang Serve Example**
```bash
python -m sglang.launch_server --model-path yulan-team/YuLan-Mini --port 30000 --host 0.0.0.0
```
---
## The Team
YuLan-Mini is developed and maintained by [AI Box, Renmin University of China](http://aibox.ruc.edu.cn/).
## License
- The code in this repository is released under the [MIT License](./LICENSE).
- Policies regarding the use of model weights, intermediate optimizer states, and training data will be announced in future updates.
- Limitations: Despite our efforts to mitigate safety concerns and encourage the generation of ethical and lawful text, the probabilistic nature of language models may still lead to unexpected outputs. For instance, responses might contain bias, discrimination, or other harmful content. Please refrain from disseminating such content. We are not liable for any consequences arising from the spread of harmful information.
## Citation
If you find YuLan-Mini helpful for your research or development, please cite [our technical report](https://arxiv.org/abs/2412.17743):
```
@misc{hu2024yulanmini,
title={YuLan-Mini: An Open Data-efficient Language Model},
author={Yiwen Hu and Huatong Song and Jia Deng and Jiapeng Wang and Jie Chen and Kun Zhou and Yutao Zhu and Jinhao Jiang and Zican Dong and Wayne Xin Zhao and Ji-Rong Wen},
year={2024},
eprint={2412.17743},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.17743},
}
```
|