MMLU is a benchmark that measures the general knowledge of language models through multiple choice questions. The name comes from Massive Multitask Language Understanding.
How it works
The test gathers questions from dozens of areas, such as history, law, medicine, math and science. Each question has alternatives, and only one is correct.
The model receives the question and chooses an alternative. The final result is the percentage of correct answers, overall and by area. The benchmark was presented in a scientific paper in 2020, and the question set is public.
The test can be applied without examples, or with a few examples in the prompt. The way it is applied changes the score, so comparisons only make sense when they follow the same method.
Practical example
A startup wants to choose a model for a study assistant. It compares the score of two models on MMLU, which indicates who has the better general knowledge base.
But the team remembers that the test is multiple choice. It does not show whether the model writes well, follows long instructions or codes. Therefore, it also consults other benchmarks and tests the models with real questions from the public.
| Information | What MMLU says |
|---|---|
| General knowledge | Yes |
| Programming ability | No |
| Writing quality | No |
What it's used for
MMLU serves as a quick measure of broad knowledge. Labs often cite the score when launching models, and researchers use it to track model evolution over time.
Limitations
With the advancement of models, many started to get high scores, which reduces the test's ability to separate a good model from an excellent one. This effect is called saturation.
There were also discussions about questions with errors and about contamination, when the model has already seen the questions during training. For this reason, the MMLU score should be read with care and alongside other evaluations.
Difference between MMLU and HumanEval
MMLU measures general knowledge with multiple choice questions. HumanEval measures the ability to write code that passes tests. A model can do well on one and average on the other.