Prompt cache stores the processing of a part of the prompt that repeats across calls. When you resend the same beginning of text, the provider reuses the work already done, which usually makes the response faster and cheaper.

How it works

A prompt in real applications has a fixed part, such as instructions, rules and documents, and a variable part, which is the user's question. Processing the fixed part every time is wasteful.

With cache, the server processes the fixed segment once and keeps the internal result for a while. On subsequent calls that start with the same content, it skips that work.

Three points show up in most providers:

  • Identical prefix: cache applies to the start of the prompt. If something changes at the beginning, reuse is lost.
  • Lifetime: cache expires after a period of no use.
  • Minimum size: some providers only apply cache to prompts above a certain size.

Details, such as pricing rules and duration, vary by provider. Check the official documentation for what you use.

Practical example

You have an assistant that answers questions based on a 30 page manual. The manual goes in every call. If you put the manual and instructions at the start of the prompt, and the question at the end, the manual can be reused from cache.

The ideal organization looks like this:

Texto
[instruções fixas]
[manual completo]
[pergunta do usuário]   <- só esta parte muda

If you put today's date at the beginning, the prefix would change every day and cache would be less useful. For that reason, variable items go at the end.

What it's for

It is used to reduce cost and decrease wait time in cases of repeated context. Code agents benefit a lot: at each step, they send the same conversation history plus a new segment, and a good part can come from cache.

It also helps reduce time to first token on long prompts.

Difference between prompt cache and semantic cache

Prompt cache reuses the processing of the beginning of the text, and the model still generates a new response. Semantic cache already stores ready responses and returns one of them when a similar question arrives, without calling the model.

Cautions

Cache does not change response quality, but it requires discipline in prompt assembly. Avoid putting variable content at the start and measure the result, looking at the usage indicators that the provider returns.