Latency is the time elapsed between sending a request to a system and receiving the response. In artificial intelligence, latency is the model's response time: how long it takes for AI to answer a question.
How it works
Latency is measured in milliseconds (ms) and depends on several factors: the distance between the client and the server, the model size, the number of tokens generated, the processing load and the network infrastructure.
Practical example
If you send a question to a language model and the model takes 200ms to generate the first word, the initial latency is 200ms. If the model generates 20 tokens at a rate of 40 tokens per second, the total latency will be 200ms plus the 500ms of generation, totaling 700ms.
What it's for
Latency is critical in interactive applications, such as chatbots, voice assistants, recommendation systems and user interfaces. Low latency means faster responses and a better user experience.
Related terms
- throughput: rate of tokens generated per second
- tempo-de-resposta: synonym for latency
- modelo-de-ia: system that generates responses
How to test today
Check the model in the official documentation, which usually lists the average latency in milliseconds per request.
Frequently asked questions
- What is the difference between latency and throughput? Latency is the response time of a request; throughput is the amount of tokens generated per second.
- Is very high latency normal? Very high latency can indicate network problems, server load or a slow model.
- How to reduce latency? Use faster models, traffic routing, cache and better connectivity.