File size: 9,245 Bytes
aaf2464 67d0528 aaf2464 67d0528 aaf2464 67d0528 aaf2464 67d0528 aaf2464 57405a4 aaf2464 57405a4 aaf2464 67d0528 aaf2464 d056075 aaf2464 d056075 aaf2464 aaab13f aaf2464 d056075 aaf2464 d056075 aaf2464 d056075 b9a0a94 67d0528 b9a0a94 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 |
---
language:
- en
license: apache-2.0
tags:
- pretrained
datasets:
- Skylion007/openwebtext
- c4
- wikimedia/wikipedia
- tiiuae/falcon-refinedweb
- izumi-lab/open-text-books
- togethercomputer/RedPajama-Data-V2
- databricks/databricks-dolly-15k
- euclaise/reddit-instruct-curated
- CohereForAI/aya_dataset
pipeline_tag: text-generation
widget:
- messages:
- role: user
content: Specs of a game about trolls and warriors in a fantasy world.
- messages:
- role: user
content: Reducing waste generation is essential to...
- messages:
- role: user
content: Water, planet, resource, future
- messages:
- role: user
content: Background story of an RPG game about wizards and dragons in a sci-fi
world. The story takes place in a...
inference:
parameters:
max_new_tokens: 250
do_sample: true
temperature: 0.65
top_p: 0.55
top_k: 35
repetition_penalty: 1.176
model-index:
- name: Minueza-32M-Base
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: AI2 Reasoning Challenge (25-Shot)
type: ai2_arc
config: ARC-Challenge
split: test
args:
num_few_shot: 25
metrics:
- type: acc_norm
value: 21.33
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: HellaSwag (10-Shot)
type: hellaswag
split: validation
args:
num_few_shot: 10
metrics:
- type: acc_norm
value: 26.39
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU (5-Shot)
type: cais/mmlu
config: all
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 24.8
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: TruthfulQA (0-shot)
type: truthful_qa
config: multiple_choice
split: validation
args:
num_few_shot: 0
metrics:
- type: mc2
value: 47.45
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: Winogrande (5-shot)
type: winogrande
config: winogrande_xl
split: validation
args:
num_few_shot: 5
metrics:
- type: acc
value: 53.2
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GSM8k (5-shot)
type: gsm8k
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 0.38
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
name: Open LLM Leaderboard
---
# Minueza-32M-Base
## Summary
Minueza-32M-Base is a foundation model with 32 million parameters trained from scratch on a large corpus of text in English.
It's available in the following formats: [Safetensors](https://huggingface.co/Felladrin/Minueza-32M-Base), [GGUF](https://huggingface.co/Felladrin/gguf-Minueza-32M-Base), and [ONNX](https://huggingface.co/Felladrin/onnx-Minueza-32M-Base).
And it's being released alongside some fine-tuned versions:
- [Minueza-32M-UltraChat](https://huggingface.co/Felladrin/Minueza-32M-UltraChat): Trained on a single conversational dataset.
- [Minueza-32M-Chat](https://huggingface.co/Felladrin/Minueza-32M-Chat): Trained on a mix of conversational datasets.
- [Minueza-32Mx2-Chat](https://huggingface.co/Felladrin/Minueza-32Mx2-Chat): Sparse Mixture of Experts trained on interleaved conversational datasets.
- [And more...](https://huggingface.co/models?other=base_model:Felladrin/Minueza-32M-Base)
## Intended Uses
This model was created with the following objectives in mind:
- Run on mobile web browsers via [Transformers.js](https://huggingface.co/docs/transformers.js).
- Run fast on machines without GPU.
- Serve as a base for fine-tunes using ChatML format, hence the two additional special tokens (`<|im_start|>` and `<|im_end|>`) with `<|im_end|>` as default EOS token.
- ChatML works great for both instruction and chat models, so if all fine-tunes are made following the ChatML pattern, other users might benefit from the easiness of creating merges.
## Datasets
The model was trained on a subset of each of the following non-synthetic datasets:
- [Skylion007/openwebtext](https://huggingface.co/datasets/Skylion007/openwebtext)
- [c4](https://huggingface.co/datasets/c4)
- [wikimedia/wikipedia - 20231101.simple](https://huggingface.co/datasets/wikimedia/wikipedia/viewer/20231101.simple)
- [tiiuae/falcon-refinedweb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)
- [izumi-lab/open-text-books](https://huggingface.co/datasets/izumi-lab/open-text-books)
- [togethercomputer/RedPajama-Data-V2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2)
- [databricks/databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
- [euclaise/reddit-instruct-curated](https://huggingface.co/datasets/euclaise/reddit-instruct-curated)
- [CohereForAI/aya_dataset - original english annotations](https://huggingface.co/datasets/CohereForAI/aya_dataset/viewer/default/train?f[language_code][value]=%27eng%27)
The subsets were interleaved to form the final training corpus of approximately 650 million tokens.
## Model Architecture
This is a transformer model with the Mistral architecture, trained on a context window of 2048 tokens.
| Configuration | Value |
| :---------------------- | :---- |
| max_position_embeddings | 2048 |
| hidden_size | 312 |
| intermediate_size | 1092 |
| num_attention_heads | 12 |
| num_hidden_layers | 10 |
| num_key_value_heads | 4 |
| vocab_size | 32002 |
The pretraining was made with these hyperparameters and frameworks:
| Hyperparameter | Value |
| :-------------------------- | :-------------------------------------------- |
| learning_rate | 5e-05 |
| train_batch_size | 1 |
| eval_batch_size | 1 |
| seed | 42 |
| gradient_accumulation_steps | 8 |
| total_train_batch_size | 8 |
| optimizer | Adam with betas=(0.9,0.999) and epsilon=1e-08 |
| lr_scheduler_type | linear |
| Framework | Version |
| :----------- | :---------- |
| Transformers | 4.38.0.dev0 |
| Pytorch | 2.1.2 |
| Datasets | 2.16.1 |
| Tokenizers | 0.15.1 |
## Usage
This is just a base model. For your task, you will likely want to perform application-specific fine-tuning as recommended above.
Also note that this model was trained on internet text data, which may contain biases, offensive or inappropriate content, and may produce incorrect or irrelevant responses. No evaluation has been conducted, so use with care.
Having that said, here's how you can run it:
```python
from transformers import pipeline
generate = pipeline("text-generation", "Felladrin/Minueza-32M-Base")
prompt = "The best way to improve your health is"
output = generate(
prompt,
max_new_tokens=256,
do_sample=True,
temperature=0.72,
top_p=0.73,
top_k=50,
repetition_penalty=1.176,
)
print(output[0]["generated_text"])
```
## [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Felladrin__Minueza-32M-Base)
| Metric |Value|
|---------------------------------|----:|
|Avg. |28.92|
|AI2 Reasoning Challenge (25-Shot)|21.33|
|HellaSwag (10-Shot) |26.39|
|MMLU (5-Shot) |24.80|
|TruthfulQA (0-shot) |47.45|
|Winogrande (5-shot) |53.20|
|GSM8k (5-shot) | 0.38|
## License
This model is licensed under the [Apache License 2.0](https://huggingface.co/Felladrin/Minueza-32M-Base/resolve/main/license.txt).
|