Prologue
The Trial of Hits and Misses
Hits and misses
You ask the cache for something. Exactly one of two things happens.
- Hit — the cache has it. You get the answer immediately. No database, no network trip to origin, no recomputation.
- Miss — the cache does not have it. You must go and get the answer the slow way, and then (usually) put a copy in the cache for next time.
That is the entire vocabulary. Hit: found it. Miss: didn't.
Hit ratio
The hit ratio is the fraction of requests that were hits.
hit ratio = hits ÷ (hits + misses)
If 1,000 requests produce 900 hits and 100 misses, the hit ratio is 900 ÷ 1000 = 0.9, or 90%.
Why it matters — work out the average response time. Say a hit takes 1ms and a miss takes 100ms:
average = (0.90 × 1ms) + (0.10 × 100ms)
= 0.9ms + 10ms
= 10.9ms
Two things stand out:
First: you went from 100ms to 10.9ms. Nearly ten times faster.
Second, and more important: almost all of the remaining time — 10ms of the 10.9ms — is the misses. At a 90% hit ratio, the 10% that miss are still doing 92% of the waiting.
Now try 99%:
average = (0.99 × 1ms) + (0.01 × 100ms) = 1.99ms
Going from 90% to 99% is another 5× improvement. The last few percent of hit ratio are worth far more than the first fifty. Keep that in mind — it comes back in Lesson 13.
We are only defining hit ratio here. How to deliberately raise it is Lesson 13.
Cold and warm
A cold cache is empty. It has just been created, or restarted, or cleared. Every request misses.
A warm cache has been running long enough to hold the things people actually ask for. Most requests hit.
Every cache starts cold. The period in between — where your app is running but the cache has not filled up yet — is genuinely dangerous, because your database is taking the full load with no protection. This is the cold start problem, and Lesson 12 will show you what an attacker can do with it.
Keys: the most important idea in this lesson
A cache stores pairs. Each pair is:
- a key — the question
- a value — the answer
To get something out, you ask with the same key you put it in with. If the key does not match exactly, it is a miss.
So the whole behaviour of your cache depends on how you write your keys. Here is the rule:
Keep this rule in mind. Missing inputs cause many of the cache bugs we'll meet.
A key that is too coarse
You cache the product price:
key: product:42:price value: 25.00
Looks fine. Then someone visits from Japan and sees ¥25.00 instead of ¥3,800. Or a wholesale customer with a negotiated discount sees the retail price. Or a logged-in member sees the non-member price.
The price depends on currency and customer type. Those inputs changed the answer, and they were not in the key. So the cache handed out an answer to a question nobody asked.
The fix:
key: product:42:price:currency=JPY:tier=wholesale
A key that is too coarse gives wrong answers. This is the worse failure of the two, because nothing crashes — it just quietly lies.
A key that is too fine
Overcorrecting, you add everything you can think of:
key: product:42:price:currency=JPY:tier=wholesale:request_id=a91f3c:time=1699384726
Every request has a different request ID and a different timestamp. So every key is unique. So every request is a miss. You have built a cache with a hit ratio of zero, and it is now slower than having no cache at all, because you also pay the cost of storing everything.
A key that is too fine gives no hits.
The judgement
Good key design sits between those two failures. Include everything that changes the answer. Include nothing else.
When you are unsure, ask: “If two requests have different values for this thing, would they get different answers?” If yes, it goes in the key. If no, it stays out.
Values: what you actually store
The value is what you get back on a hit. Some choices:
How much to store. You can cache the whole rendered product page, or the product record, or just the price. Bigger values mean fewer, larger cache entries; smaller values mean more entries and more flexibility. A cached whole page is fast but must be thrown away when anything on it changes. A cached price survives a description edit.
Whole objects or references. You can store the full review objects, or just a list of review IDs and look them up separately. Storing IDs keeps entries small and avoids duplicating the same review in twenty places — at the cost of a second lookup.
Serialization. Most caches store bytes, not objects. Serialization is turning an object in your program's memory into a sequence of bytes that can be stored or sent, and deserialization is turning it back. This is not free. For small, very hot values, the time spent serializing can be a real fraction of the time you saved — which is one reason keeping a cache inside your own process (Lesson 5) is so fast: no serialization at all.
TTL: how long a copy is allowed to live
TTL stands for time to live. It is the number of seconds a cached copy is considered usable. When the TTL runs out, the copy has expired, and the next request for it is treated as a miss.
key: product:42:price:currency=USD value: 25.00 TTL: 300 ← usable for 5 minutes, then thrown away
TTL is your main tool for controlling staleness. A 300-second TTL is you saying: “I accept that this price may be up to five minutes out of date.”
That is a business decision as much as a technical one. Five minutes of a stale product description is fine. Five minutes of a stale stock count might mean selling something you do not have.
TTL is introduced here because you need it from Lesson 3 onward. The policies for what to throw away when memory runs out are Lesson 10 — a related but genuinely different topic.
Warming: filling the cache on purpose
Cache warming is loading data into a cache before anyone asks for it, so the first real user gets a hit instead of a miss.
Three common approaches:
Warm on deploy. After releasing new code, run a script that requests your most important pages so their entries exist before traffic arrives.
Warm from a log. Take the top few thousand keys from yesterday's traffic and load those. Simple and effective, because yesterday's popular items are usually today's.
Lazy warming. Do nothing special — let the cache fill as real requests arrive. This is the default. It is fine for small caches and dangerous for large ones, because the whole warm-up period runs with your database unprotected.
Warming matters most when your cache is large and your database is only just able to cope without it. If your database can handle full traffic with a cold cache, warming is a nicety. If it cannot, warming is the difference between a deploy and an outage.
“The cache key is just the record's ID.”
The ID is the beginning of the key. The rest is language, currency, permissions, device, feature flags — every input that changes the answer. The forgotten one is where your bug lives.
In a room far from the light, the Latency Lord watches a single request pass by. It asks for product 999999999 — a product that does not exist and never has.
He watches it slip past the browser, which has no copy of it. Past the edge, which has no copy of it. Past the application cache, which has no copy of it. And he watches it strike the database directly, and the database dutifully search its entire index for something that was never there.
He has found the weakness: nonexistent products are not cached, so every new fake ID forces another useless database lookup. He begins generating thousands of fake product IDs.