How it works
Greedy decoding is a strategy for generating a sequence by choosing, at each step, the token with the highest probability among the available options. A token can correspond to a word, part of a word, space or punctuation mark. The model calculates a probability distribution for the next token, and the algorithm picks the most likely value at that moment.
After the choice, the token is added to the text and the model calculates the next distribution considering what has already been written. The process continues until it finds an end token or reaches a defined limit. Because the choice is always the highest local probability, the method is called greedy.
The strategy does not search all possible sequences nor evaluate the complete result before choosing. This makes the logic simple, but a local decision can lead to a less suitable continuation later. In text tasks, the most probable option now does not always produce the most coherent response as a whole.
Greedy decoding is also not exactly the same as setting a low temperature. Temperature changes the distribution used in sampling; greedy decoding selects the highest value directly. Different systems can combine generation controls in different ways, so check the model's or API's documentation.
Practical example
Suppose that, after the sentence "A capital do Brasil é", the model assigns higher probability to "Brasília" than to the other options. Greedy decoding chooses "Brasília" and then calculates the next token. If another choice is slightly more probable in that context, it wins, even if a complete alternative sequence could be better.
In a simplified writing example, the first token with the highest probability can start a common sentence, but limit the response to a generic formulation. The algorithm does not go back to compare that response with other sequences. This does not mean the result will always be bad: for predictable outputs and controlled tasks, direct choice can work well.
When testing an application, keep the same input and observe if the strategy produces the desired stability. Also compare text quality, repetition and instruction following. A reproducible output is not a synonym for a correct output.
What it is for
Greedy decoding is useful when you want a simple and predictable selection rule for generating text or other sequential data. It also helps explain how a model transforms probabilities into concrete tokens.
The method can be inadequate when the task requires diversity or when a bad local choice harms the continuation. To generate varied alternatives, systems can use sampling and controls like top-k, top-p or temperature. To explore several continuations, other techniques can keep more than one hypothesis under consideration.
In practice, do not choose a strategy by name alone. Test it with representative examples and evaluate if the response is correct, useful and stable for the product's goal. In factual content, selecting the most probable token does not verify the truth of the statement; external sources and validations remain necessary.