How it works
In AI, scaling involves hardware and software. On hardware: more GPUs, more servers, more memory. On software: load balancing, cache, request queues and batching.
Services like vLLM and model APIs use continuous batching techniques to process multiple requests at the same time. This increases throughput without increasing latency per request.
There are two types of scale. Vertical scale: using a more powerful GPU or server. Horizontal scale: adding more instances. Vertical scale has a physical limit. Horizontal scale is more flexible, but requires orchestration.
Practical example
A startup launching an AI chatbot can start with a single GPU running an open source model. As users grow, it needs to scale. First, it optimizes the code (batching, cache). Then, it adds more instances with a load balancer. If the model is too large, it can use tensor parallelism to split the model across multiple GPUs.
Cost grows with scale. An H100 GPU can cost thousands of dollars per month. For that reason, many companies use model APIs (OpenAI, Anthropic) instead of running their own models, outsourcing scale.
What it's for
Plan infrastructure, estimate costs and avoid bottlenecks as the application grows. Understanding scalability helps decide between running your own model or using an API, and designing systems that do not break under load.
Difference between scalability and elasticity
Scalability is the ability to grow. Elasticity is the ability to grow and shrink automatically according to demand. An elastic system scales up during usage peaks and down during low demand periods, saving costs.