How it works
Input tokens are the units of text or other data sent to the model in a request. They can include the instruction, the question, the conversation history and document snippets. In multimodal models, the input can also contain images, audio or other formats, as accepted by the service.
Output tokens are the units produced by the model in response. In a chat, they normally form the text that the person reads. In a tool call or a structured response, they can compose arguments, data fields or content intended for another step of the system.
A token is not always equivalent to a word. Text is split into units by the model's tokenizer. Common words can become one token or several, while punctuation, spaces and parts of words can also take up tokens. For this reason, the exact number depends on the language, the content and the tokenizer used.
A request can have separate limits for input and output, in addition to a total context limit. The system needs to consider the included history and the generated response. Cutting old messages or limiting the response can help keep the call within limits, but it can remove useful information.
Practical example
Consider an assistant that receives a question and a documentation snippet to answer. The instruction, the question and the snippet form part of the input. The explanation returned by the model is output. If the system also includes previous messages, those messages increase the material sent, even if the person did not type them in the current request.
In a programming flow, the input can contain the task, the relevant code and the error message. The response can bring an explanation or a proposed change. If the tool sends the entire repository without need, it increases the input volume and can make it harder for the model to focus on the important files.
An implementation can log counts reported by the provider, when available, and track limits and consumption. It can also summarize history, retrieve only related documents or request a shorter response. These decisions should preserve the context necessary for the task, not just reduce numbers without criteria.
What it's for
Distinguishing input and output tokens helps design integrations, control request size and understand usage reports. It also allows diagnosing why a call exceeded the context limit or why an application received an incomplete response.
The difference matters when comparing APIs. A provider can report counts and its own rules, and details vary between models. The developer should consult the documentation of the model used, instead of assuming that character or word count represents exactly the token count.
This separation also helps in product design. A long conversation can use many input tokens because of history, while a long generation increases output tokens. Summaries, context selection and response limits are architecture options, but each involves a trade off between conciseness and available information.
To interpret the numbers correctly, record which model and request were used and differentiate what was sent from what was generated. Tokens are a measure of the model's textual processing, not a direct measure of the difficulty of the question or the quality of the answer. Quality evaluation requires its own criteria, such as correctness, usefulness and adherence to instructions.