We all love the word “cache.” In software engineering it usually means “free speed.” You store a database query in Redis and suddenly your app runs fast and your server bill drops. It feels like magic.
So when Anthropic announced “Prompt Caching” for Claude 3.5 Sonnet and Haiku developers cheered. The promise was simple. If you send the same massive text block (like a book or a codebase) over and over again you get a ninety percent discount.
But this is not Redis. This is an LLM.
If you blindly turn on caching without doing the math you might wake up to an invoice that is twenty five percent higher than usual. There is a hidden tax on this feature and it relies on a very aggressive timer.
Here is how to avoid the “Cache Trap” and actually save money.
The Math Behind the Trap
Standard caching logic tells us that writing to cache is cheap or free. With Anthropic caching is a premium service.
You pay a “Write Cost” to upload your prompt into their short term memory. This Write Cost is 25 percent higher than the standard input token price.
Standard Input
$3.00 per million tokens (Claude 3.5 Sonnet)
Cache Write
$3.75 per million tokens
Cache Read
$0.30 per million tokens
Do you see the gamble? You are paying a premium upfront betting that you will reuse that text enough times to recover the investment. If you send a prompt once and never use it again you just volunteered to pay a twenty five percent tax for no reason.
The Five Minute Rule
This is the silent killer.
Anthropic’s ephemeral cache has a Time To Live (TTL) of 5 minutes.
This is not like a browser cache that stays for days. It is a “use it or lose it” system. Every time you hit the cache the timer resets to five minutes. But if your user walks away for a coffee and comes back six minutes later the cache is gone.
The Loss Scenario
Imagine you have a chatbot with low traffic.
- 10:00 AM: User asks a question. You pay $3.75 (Write Cost).
- 10:06 AM: User asks a follow up. The cache expired one minute ago. You pay $3.75 (Write Cost) again.
Total Cost: $7.50.
Cost without caching: $6.00.
You just lost money by trying to be smart. For caching to work you need “high frequency” traffic or a very engaged user who types fast.
The Structure Problem
The cache is fragile. It works on “prefixes.”
To hit the cache your new request must match the cached text exactly from the start.
Bad Structure
System Prompt -> User Name (Dynamic) -> Big Document (Static)
Because the “User Name” changes every time the API sees the “Big Document” as new data. It cannot cache the end of the prompt if the middle changes.
Good Structure
System Prompt -> Big Document (Static) -> User Name (Dynamic)
You must push all your heavy static content to the absolute front of the prompt. If you put a timestamp or a request ID at the top you break the cache for everyone.
The Minimum Viable Token Count
You cannot cache a “Hello World” message.
Anthropic enforces a minimum block size to enable caching. For Claude 3.5 Sonnet you need at least 1024 tokens (roughly 750 words). For Haiku it is often higher (2048 tokens).
If your system prompt is short do not bother adding the cache_control flag. You might think you are optimizing but the API will likely ignore it or worse charge you for overhead you are not using.
When to Actually Use It
Despite the risks the savings are real if you fit the profile. The “Break Even” point is surprisingly low.
If you hit the cache just one time after writing it you save money.
The Math:
- Standard: $3 (Call 1) + $3 (Call 2) = $6.
- Cached: $3.75 (Write) + $0.30 (Read) = $4.05.
You save thirty percent instantly on the second call. By the tenth call you are paying pennies.
The Verdict
Use Prompt Caching if:
- You are building an “Agent” loop where the AI talks to itself ten times in a row.
- You have a “Analyze this Book” feature where the user asks many questions about one document.
- You are running high volume evaluations on a fixed dataset.
Do not use Prompt Caching if:
- Your users are slow (response times > 5 minutes).
- Your prompt has dynamic variables at the start.
- You are just trying to optimize a short 500 token system prompt.
Caching is a powerful weapon but like all weapons it kicks back if you hold it wrong. Check your logs. If your “Cache Hit Rate” is under fifty percent turn it off.
