Spaces:

AtlaAI

/

judge-arena

like 85

Running

After spending more time on this leaderboard, I completely agree. If both models are saying 4/5 or 5/5 for something I think is a 2/5, there is no mechanism to flag/penalize the models for this and it happens frequently enough for me to notice.

This is potentially an issue with your new Atla-8b-Preview btw

Hey @pszemraj

Thanks for voting on Judge Arena; excited to see that you're engaging with our platform! I'll give you my quick 30s rant on "both bad" options:

"Vote A", "Vote B", or "Tie" are mutually exclusive, and collectively exhaustive (MECE). Once you add “both bad,” you open the door to “both good,” “both mediocre,” and so on. That extra complexity muddies the scoring system and forces more granular, time-consuming judgments. By sticking to A/B/Tie, we keep things logically coherent, maintain clarity, and encourage decisive feedback without scattering options.