| license: mit | |
| tags: | |
| - nlp | |
| - math | |
| language: | |
| - en | |
| pipeline_tag: text-generation | |
| ## Rho-1: Not All Tokens Are What You Need | |
| The Rho-1 series are pretrained language models that utilize Selective Language Modeling (SLM) objectives. | |
| In math reasoning pretraining, SLM improves average few-shot accuracy on GSM8k and MATH by over 16%, achieving the baseline performance 5-10x faster. | |
| For more details please check our [github](https://github.com/microsoft/rho). | |