How it works
Cost per token is how a service expresses how much it charges for processing text split into tokens. A token is a unit of text used by the model. It can correspond to a short word, part of a word, punctuation or another fragment, depending on the language and tokenizer.
In model APIs, billing usually separates tokens sent by the user from tokens generated in the response. Some offerings also distinguish cached processing, tool calls, images, audio or other resources. The exact rule depends on the provider and the model, so the official pricing page and the API documentation should be consulted before estimating spend.
To calculate a call, count or estimate the input and output tokens, apply the corresponding rate for each category and sum the values. The estimate needs to consider system instructions, included history, retrieved documents and the format required in the response, not just the question visible to the user.
A long conversation can repeat instructions and previous excerpts in several calls. If the application sends the entire history again, those tokens remain part of the input, even if the person does not type them again. Therefore, context architecture and output limits affect consumption as much as the model choice.
Practical example
Imagine a support assistant that receives a question, queries a knowledge base and responds in a few sentences. The input can include the question, the assistant instructions, the recent history and the excerpts found in search. The response also consumes tokens, according to the size produced.
To compare two configurations, use the same representative questions and record input and output tokens separately. If a configuration uses a different model, a larger response or more context, also compare quality and usefulness. The lowest isolated consumption does not mean better cost benefit if the response requires several attempts or additional review.
A simple spreadsheet can have columns for call type, input volume, output volume, model chosen and calculated value according to the current official table. Avoid storing estimates as fixed values in code: rates and rules can change. Read the provider source when updating the application.
What it is for
This metric helps design an application's budget, compare models for a task and investigate unexpected increases in API usage. It also allows discussing efficiency with observed data, instead of assuming all requests have similar size.
To optimize, the team can reduce irrelevant context, limit response length, reuse results when it is safe and route simple tasks to less costly configurations. Each change should be tested against the expected quality and privacy needs.
Cost per token is not the same as total product cost. Own infrastructure, storage, search, external tools, monitoring, development and support can also generate expenses. In a hosted solution, confirm which items are billed separately and which limits apply.
Track consumption per user, task or environment without recording unnecessary sensitive data. This helps detect repeated calls and excessive contexts, keeping the analysis focused on operational metrics and not on the private content of each conversation.