HumanEval is a set of Python programming problems published by OpenAI alongside the paper on the Codex model. It has become one of the most cited references when someone wants to compare AI models that write code.

How it works

Each problem brings a function with a name, signature and a text description, called a docstring. The model receives this start and needs to complete the function body.

Then, the generated code runs against hidden unit tests. If all pass, the problem counts as solved. If any test fails, it counts as an error.

The most common result is pass@1: the chance the model gets it right on the first try. There is also pass@k, which allows k attempts and checks if at least one passes.

The advantage of this method is that it does not depend on opinion. The code works or it does not work, and anyone can repeat the test.

Practical example

Imagine a problem like this: write a function that receives a list of numbers and says if two of them are closer than a given limit.

Python
def has_close_elements(numbers, threshold):
    """Retorna True se existirem dois números a menos de threshold de distância."""

The model writes the body. The tests then run with different lists, including edge cases like an empty list. If the answer works in all of them, that problem scores a point.

When you see a model announced with a percentage on HumanEval, that is what is behind it: the fraction of problems of this type that it solved.

What it is for

HumanEval is used to have a simple and repeatable number for code generation quality. Anyone who chooses a model for a programming assistant uses this kind of score as a first filter.

It also helps teams that train or fine tune models to see if a change improved or worsened programming ability.

Limits of HumanEval

The problems are small, single function, and the set is public. This means a model may have seen similar examples in training, which inflates the score.

Real programming work involves multiple files, libraries and project context. For that reason HumanEval does not replace tests on your own code.

For tasks closer to day to day, like fixing issues in real repositories, see SWE-bench. Use both scores together and, whenever possible, do your own test with a case from your project.