U-MATH and μ-MATH - University-level math evaluation
Collection
Paper: A UNIVERSITY-LEVEL BENCHMARK FOR EVALUATING MATHEMATICAL SKILLS IN LLMS
•
4 items
•
Updated
•
15
Toloka
company
Verified
AI & ML interests
Human In The Loop - data labeling, model training and hosting, human verification, and more
Recent Activity
View all activity
toloka's activity
k4black
updated
a
collection
8 days ago
cogwheelhead
posted
an
update
about 1 month ago
Post
286
Hey!
Me and my team recently released two benchmarks on university-level math: U-MATH (for University-MATH) and μ-MATH (for Meta U-MATH).
We're working a lot on complex reasoning for LLMs, and we were in particular interested in evaluating university-curricula math skills — in topics such as differential calculus and linear algebra — for their wide applicability and practicality.
We noticed that available benchmarks at the time were either at or below high-school level, or mainly leaning towards Olympiad-style problems, or synthetically generated from a set of templates / seeds.
We wanted focus on university curricula and we wanted "organic" variety, so we created our own benchmark using problems sourced from actual teaching materials used in top US universities — that is how U-MATH came to be.
We also, and that is my primary focus in particular, are very eager on studying and improving evaluations themselves, since the standard llm-as-a-judge approach is known to be noisy and biased, but that often remains unaccounted for. So we then created a U-MATH-derived benchmark to do "meta-evaluations" — i.e. evaluate the evaluators — which allows to quantify their error-rates, study their behaviors and biases, and so on.
I'm super excited to be sharing those publicly!
toloka/u-math
toloka/mu-math
Me and my team recently released two benchmarks on university-level math: U-MATH (for University-MATH) and μ-MATH (for Meta U-MATH).
We're working a lot on complex reasoning for LLMs, and we were in particular interested in evaluating university-curricula math skills — in topics such as differential calculus and linear algebra — for their wide applicability and practicality.
We noticed that available benchmarks at the time were either at or below high-school level, or mainly leaning towards Olympiad-style problems, or synthetically generated from a set of templates / seeds.
We wanted focus on university curricula and we wanted "organic" variety, so we created our own benchmark using problems sourced from actual teaching materials used in top US universities — that is how U-MATH came to be.
We also, and that is my primary focus in particular, are very eager on studying and improving evaluations themselves, since the standard llm-as-a-judge approach is known to be noisy and biased, but that often remains unaccounted for. So we then created a U-MATH-derived benchmark to do "meta-evaluations" — i.e. evaluate the evaluators — which allows to quantify their error-rates, study their behaviors and biases, and so on.
I'm super excited to be sharing those publicly!
toloka/u-math
toloka/mu-math
k4black
authored
a
paper
about 2 months ago
tilgasergey
authored
2
papers
about 2 months ago
cogwheelhead
authored
a
paper
about 2 months ago
k4black
updated
a
dataset
about 2 months ago
k4black
updated
a
collection
about 2 months ago
k4black
updated
a
dataset
about 2 months ago
cogwheelhead
updated
2
datasets
about 2 months ago
KatyaA
updated
2
datasets
2 months ago
KatyaA
authored
a
paper
6 months ago
KatyaA
authored
4
papers
7 months ago
NEREL: A Russian Dataset with Nested Named Entities, Relations and Events
Paper
•
2108.13112
•
Published
TAPE: Assessing Few-shot Russian Language Understanding
Paper
•
2210.12813
•
Published
RuCoLA: Russian Corpus of Linguistic Acceptability
Paper
•
2210.12814
•
Published
•
1
Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models
Paper
•
2202.07791
•
Published