File size: 2,768 Bytes
a47eb6a 281ce79 e574d26 a47eb6a e574d26 3f30ef0 e574d26 49ae096 e574d26 49ae096 e574d26 49ae096 dde2182 49ae096 3f30ef0 49ae096 281ce79 49ae096 efc82e0 dde2182 49ae096 e574d26 dde2182 e574d26 49ae096 e574d26 49ae096 e574d26 dde2182 e574d26 3f30ef0 e574d26 49ae096 e574d26 49ae096 e574d26 49ae096 e574d26 dde2182 e574d26 49ae096 e574d26 49ae096 e574d26 49ae096 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 |
---
license: openrail
datasets:
- teknium/OpenHermes-2.5
language:
- en
library_name: transformers
pipeline_tag: text-generation
---
# Model Card for neoncortex/mini-mistral-openhermes-2.5-chatml-test
A tiny Mistral model trained as an experiment on teknium/OpenHermes-2.5.
## Model Details
A 63M parameter auto-regressive LM using Mistral architecture as a base.
- Multi-query Attention instead of Grouped-query Attention.
- Sliding window is disabled.
- Modified ChatML instead of Mistral chat template - TL;DR I used '<|im_start|>human' instead of '<|im_start|>user'
### Model Description
Just doing it to see what happens.
It'll take about 40 to 45 hours to train on two Nvidia RTX 3060 12GB.
It uses ChatML for the chat template, but I messed up the template in the dataset,
using '<|im_start|>human' instead of '<|im_start|>user'. ¯\_(ツ)_/¯
So, here's the bits:
```
{%- set ns = namespace(found=false) -%}
{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
{%- set ns.found = true -%}
{%- endif -%}
{%- endfor -%}
{%- for message in messages %}
{%- if message['role'] == 'system' -%}
{{- '<|im_start|>system\n' + message['content'].rstrip() + '<|im_end|>\n' -}}
{%- else -%}
{%- if message['role'] == 'human' -%}
{{-'<|im_start|>human\n' + message['content'].rstrip() + '<|im_end|>\n'-}}
{%- else -%}
{{-'<|im_start|>assistant\n' + message['content'] + '<|im_end|>\n' -}}
{%- endif -%}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{-'<|im_start|>assistant\n'-}}
{%- endif -%}
```
- **Developed by:** RoboApocalypse
- **Funded by:** RoboApocalypse
- **Shared by:** RoboApocalypse
- **Model type:** Mistral
- **Language(s) (NLP):** English, maybe others idk
- **License:** OpenRAIL
### Model Sources
Exclusively available right here on HuggingFace!
- **Repository:** https://huggingface.co/neoncortex/mini-mistral-openhermes-2.5-chatml-test
## Uses
None
### Out-of-Scope Use
This model won't work well for pretty much everything, probably.
#### Preprocessing
Format the OpenHermes 2.5 dataset with ChatML.
#### Training Hyperparameters
- **Training regime:** bf16 mixed precision
## Evaluation
I tried to run evals but the eval suite just laughed at me.
## Model Examination
Don't be rude.
## Environmental Impact
- **Hardware Type:** 2 x NVIDIA RTX 3060 12GB
- **Hours used:** ~45 x 2.
- **Carbon Emitted:** [TBA]
### Compute Infrastructure
I trained it on my PC with no side on it because I like to watch the GPUs do their work.
#### Hardware
2 x Nvidia RTX 3060 12GB
#### Software
The wonderful free stuff at HuggingFace (https://huggingface.co)[https://huggingface.co]: transformers, datasets, trl
|