Gemma-2-Ataraxy-Gemmasutra-9B-slerp

💻 Usage

!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "recoilme/Gemma-2-Ataraxy-Gemmasutra-9B-slerp"
messages = [{"role": "user", "content": "What is a large language model?"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	29.87
IFEval (0-Shot)	76.49
BBH (3-Shot)	42.25
MATH Lvl 5 (4-Shot)	1.74
GPQA (0-shot)	10.74
MuSR (0-shot)	12.39
MMLU-PRO (5-shot)	35.63

Open Portuguese LLM Leaderboard Evaluation Results

Detailed results can be found here and on the 🚀 Open Portuguese LLM Leaderboard

Metric	Value
Average	73.97
ENEM Challenge (No Images)	75.65
BLUEX (No Images)	64.26
OAB Exams	53.76
Assin2 RTE	93.21
Assin2 STS	80.91
FaQuAD NLI	77.39
HateBR Binary	87.61
PT Hate Speech Binary	66.84
tweetSentBR	66.14