Running
132
🎵
Jaahas Jaahas
jaahas
·
AI & ML interests
None yet
Recent Activity
liked
a Space
19 days ago
r3gm/Ultimate-Vocal-Remover-WebUI
new activity
4 months ago
barisaydin/ollama:Create scripts/
new activity
4 months ago
barisaydin/ollama:Create scripts/
Organizations
None yet
jaahas's activity
liked
a
Space
19 days ago
Create scripts/
#1 opened 4 months ago
by
jaahas
Create scripts/
#1 opened 4 months ago
by
jaahas
liked
a
Space
5 months ago
liked
3
Spaces
6 months ago
updated
2
models
6 months ago
updated
a
dataset
6 months ago
updated
a
model
6 months ago
reacted to
clefourrier's
post with 🤯
10 months ago
Post
2223
Fun fact about evaluation, part 2!
How much do scores change depending on prompt format choice?
Using different prompts (all present in the literature, from
10 points for a single model!
Keep in mind that we only changed the prompt, not the evaluation subsets, etc.
Again, this confirms that evaluation results reported without their details are basically bullshit.
Prompt format on the x axis, all these evals look at the logprob of either "choice A/choice B..." or "A/B...".
Incidentally, it also changes model rankings - so a "best" model might only be best on one type of prompt...
How much do scores change depending on prompt format choice?
Using different prompts (all present in the literature, from
Prompt question? to Question: prompt question?\nChoices: enumeration of all choices\nAnswer: ), we get a score range of...10 points for a single model!
Keep in mind that we only changed the prompt, not the evaluation subsets, etc.
Again, this confirms that evaluation results reported without their details are basically bullshit.
Prompt format on the x axis, all these evals look at the logprob of either "choice A/choice B..." or "A/B...".
Incidentally, it also changes model rankings - so a "best" model might only be best on one type of prompt...