GPQA is a set of multiple choice questions in biology, physics and chemistry, written by PhD-level specialists. The idea is that they are so hard that not even a quick internet search solves them, hence the name Google-Proof.
How it works
Each question has four alternatives. Specialists in the area get it right a lot, but smart people from other areas, even with internet access, get it right much less. This design tries to ensure the question requires real knowledge and reasoning.
To test a model, you present the question with the alternatives and see if it picks the correct one. The score is the percentage of correct answers.
The subset called Diamond brings together the questions where specialists agreed on the answer and non-specialists got it wrong. That is why it is the most cited.
Like any benchmark score, the result depends on test details, such as prompt type and use of step-by-step reasoning.
Practical example
A company launches a new model and says it reached a certain percentage on GPQA Diamond. You want to know if that matters for your case.
If your product is a code assistant for a web app, GPQA says little. If it is a tool that helps researchers read scientific papers, the test is more relevant.
A good practice is to build 20 questions from your domain and compare models with them.
pergunta; resposta esperada; resposta do modelo A; resposta do modelo BWhat it is for
GPQA is used to measure advanced scientific reasoning in language models. It is one of the metrics used to compare top models.
For those choosing a model, it is one signal among many. Use it together with code tests, like SWE-bench, and with your own cases.
Caveats when reading the score
Public benchmarks can become saturated, when several models get close to the ceiling, and they can leak into training data. So always check the source and date of the result before drawing conclusions.