Throughput is the rate of units processed per unit of time, usually measured in tokens per second, requests per minute or items completed per hour. In artificial intelligence, throughput indicates the amount of tokens the model generates per second.

How it works

Throughput is measured in tokens per second, requests per minute or use cases per hour, depending on the use case. It depends on: model size, number of GPUs, network latency, batch size and infrastructure optimization. Larger models and higher latency reduce throughput, while larger GPUs and better infrastructure increase the processing rate.

Practical example

A model that generates 50 tokens per second generates 3,000 tokens per minute. A system that serves 100 users per minute with a model that takes 2 seconds per request can handle 30 requests per minute (rate limited). If the success rate is 80%, only 24 requests per minute are processed successfully.

What it's for

Throughput is critical for: high demand chatbots, batch data processing systems, AI servers, web applications and AI APIs. A high throughput rate means the system can serve more users with fewer resources, but it can increase model load and latency.

Related terms

  • latency: response time of a request
  • tokens per second: throughput measure of language models
  • modelo-de-ia: system that generates responses

How to test today

Measure the model's throughput in the documentation or by running a benchmark script, sending N requests to the model and logging the completion rate and token rate, for example, with a parallel request every 2 seconds.

Frequently asked questions

  • What is throughput in AI? It is the rate of units processed per unit of time, usually tokens per second for language models.
  • What is the difference between throughput and latency? Throughput is the amount of work done per unit of time; latency is the response time of a request. They are complementary metrics: a model can have high throughput but high individual latency.
  • How to increase throughput? Use smaller or less complex models, increase batch size, use multiple servers, optimize network infrastructure, align the processing architecture and use optimized inference.