A course on caching — what it is, where copies live, and what it costs to keep them. We follow one product page, with seven parts that need different caching rules.
Every lesson uses the same product page. Its labels show the seven answers that change at seven different speeds.
The Wayfinder collection · Vol. I
A hand-bound guide to the roads, rivers and hidden passages connecting every realm.
Hardcover Illustrated field edition
An illustrated storefront for this course. Checkout is not available.
These seven labels are the answers we will follow through the course. Most describe the same product for everyone. Part 7—the cart badge—is different: it belongs to one user, so it must never enter a shared cache.
Open the storefront's product page. It takes 900 milliseconds to appear.
Now open it again. It takes 900 milliseconds again.
Nothing changed between those two visits. The product has the same name, the same description, the same picture. Your server did the exact same work, twice, and got the exact same answer, twice.
That is the waste. Caching is the art of noticing it.
A cache (say it “cash”) is a copy of an answer, kept somewhere faster to reach than the place the answer came from.
That is the whole idea. Everything else in this course is detail about where you keep the copy, how long you keep it, and what happens when the copy stops being true.
Almost all slowness comes from one of two causes.
1. Distance. The data is far away and has to travel.
2. Recomputation. The work was already done once, and you are doing it again.
A cache fixes both, in the same way: keep the answer nearby, and stop redoing the work.
These numbers are approximate, and they vary by hardware. What matters is not the exact figures but the gaps between them.
| Where the data is | Roughly how long it takes |
|---|---|
| In your program's own memory | 100 nanoseconds |
| On a local SSD | 100 microseconds (1,000× slower) |
| On another server in the same datacentre | 500 microseconds (5,000× slower) |
| On a server on another continent | 150 milliseconds (1,500,000× slower) |
Those numbers are too small to feel. So here is the same table again, with every figure multiplied by the same amount.
These are not real durations. Only the ratios are real. If reading from memory took 1 second, then:
| Where the data is | If memory took 1 second |
|---|---|
| Program memory | 1 second |
| Local SSD | 17 minutes |
| Same datacentre | 1.4 hours |
| Another continent | 17 days |
This is the single most useful intuition in caching. Moving an answer from “another continent” to “program memory” is not a small improvement. It is the difference between waiting 17 days and waiting 1 second.
Caching gives you speed, but it charges you in freshness and memory.
| You want | What caching does | The cost |
|---|---|---|
| Speed | Reuses an answer instead of producing it again | — |
| Freshness | May return an older copy | The answer may be out of date |
| Low memory use | Stores additional copies | Every copy takes space |
A cached copy that no longer matches the latest answer is called stale.
Caching buys speed by accepting some risk of stale data and using additional memory. Every technique in this course makes that trade-off differently.
Caching is sometimes described as an optimisation added after a system becomes slow. That is only partly true.
Adding a cache creates another copy of your data. That copy may fall behind the original.
For a product description, that may be harmless. For a price or stock count, it may cause a customer to see incorrect information.
You do not need to build every cache from the beginning. But you do need to decide which answers are allowed to be old and for how long. That is a correctness decision, so it belongs in the system design.
Put the page beside its freshness promises. The same seven fields now become seven different correctness decisions.
The Wayfinder collection · Vol. I
A hand-bound guide to the roads, rivers and hidden passages connecting every realm.
Hardcover Illustrated field edition
An illustrated storefront for this course. Checkout is not available.
Each limit reflects what happens if the answer is out of date.
| # | Part | May it be stale? | If it is, what happens |
|---|---|---|---|
| 1 | Asset bundle | Indefinitely | Nothing — version the file |
| 2 | Product images | For days | An old photo remains |
| 3 | Product details | For hours | A correction appears late |
| 4 | Price | For seconds | A customer sees an old price |
| 5 | Stock count | Barely, if at all | You may oversell |
| 6 | Reviews | For minutes | A new review appears late |
| 7 | Cart badge | Not applicable | Another user's answer is wrong |
Look at each part and ask two questions.
What does it cost to produce? How long does it take, and how many database queries does it need? That tells you what is worth caching. This is The Sight — you cannot decide what to cache until you know what is expensive.
How old is it allowed to be? That tells you what you are permitted to cache, which is a different question entirely.
Look at part 1 again — the CSS and JavaScript files. It is drawn below the storefront because it is not really part of this page.
It is part of every page.
The same bundle is used by the home page, the category listing, the search results, the checkout, and the account screen. Cache it once and you have not made one page faster — you have made the whole site faster, for every visitor, on every page after their first.
The same goes for anything that repeats: the header, the navigation menu, the footer, the category list that fills the menu.
A product description is reused by everyone who views that product. A logo is reused by everyone who views anything.
And the counter-example is on the same page. The cart badge appears on every page too, exactly like the header does. But “on every page” and “the same for everyone” are different things — and only the second one makes something safe to share.
You ask the cache for something. Exactly one of two things happens.
That is the entire vocabulary. Hit: found it. Miss: didn't.
The hit ratio is the fraction of requests that were hits.
hit ratio = hits ÷ (hits + misses)
If 1,000 requests produce 900 hits and 100 misses, the hit ratio is 900 ÷ 1000 = 0.9, or 90%.
Why it matters — work out the average response time. Say a hit takes 1ms and a miss takes 100ms:
average = (0.90 × 1ms) + (0.10 × 100ms)
= 0.9ms + 10ms
= 10.9ms
Two things stand out:
First: you went from 100ms to 10.9ms. Nearly ten times faster.
Second, and more important: almost all of the remaining time — 10ms of the 10.9ms — is the misses. At a 90% hit ratio, the 10% that miss are still doing 92% of the waiting.
Now try 99%:
average = (0.99 × 1ms) + (0.01 × 100ms) = 1.99ms
Going from 90% to 99% is another 5× improvement. The last few percent of hit ratio are worth far more than the first fifty. Keep that in mind — it comes back in Lesson 13.
We are only defining hit ratio here. How to deliberately raise it is Lesson 13.
A cold cache is empty. It has just been created, or restarted, or cleared. Every request misses.
A warm cache has been running long enough to hold the things people actually ask for. Most requests hit.
Every cache starts cold. The period in between — where your app is running but the cache has not filled up yet — is genuinely dangerous, because your database is taking the full load with no protection. This is the cold start problem, and Lesson 12 will show you what an attacker can do with it.
A cache stores pairs. Each pair is:
To get something out, you ask with the same key you put it in with. If the key does not match exactly, it is a miss.
So the whole behaviour of your cache depends on how you write your keys. Here is the rule:
Keep this rule in mind. Missing inputs cause many of the cache bugs we'll meet.
You cache the product price:
key: product:42:price value: 25.00
Looks fine. Then someone visits from Japan and sees ¥25.00 instead of ¥3,800. Or a wholesale customer with a negotiated discount sees the retail price. Or a logged-in member sees the non-member price.
The price depends on currency and customer type. Those inputs changed the answer, and they were not in the key. So the cache handed out an answer to a question nobody asked.
The fix:
key: product:42:price:currency=JPY:tier=wholesale
A key that is too coarse gives wrong answers. This is the worse failure of the two, because nothing crashes — it just quietly lies.
Overcorrecting, you add everything you can think of:
key: product:42:price:currency=JPY:tier=wholesale:request_id=a91f3c:time=1699384726
Every request has a different request ID and a different timestamp. So every key is unique. So every request is a miss. You have built a cache with a hit ratio of zero, and it is now slower than having no cache at all, because you also pay the cost of storing everything.
A key that is too fine gives no hits.
Good key design sits between those two failures. Include everything that changes the answer. Include nothing else.
When you are unsure, ask: “If two requests have different values for this thing, would they get different answers?” If yes, it goes in the key. If no, it stays out.
The value is what you get back on a hit. Some choices:
How much to store. You can cache the whole rendered product page, or the product record, or just the price. Bigger values mean fewer, larger cache entries; smaller values mean more entries and more flexibility. A cached whole page is fast but must be thrown away when anything on it changes. A cached price survives a description edit.
Whole objects or references. You can store the full review objects, or just a list of review IDs and look them up separately. Storing IDs keeps entries small and avoids duplicating the same review in twenty places — at the cost of a second lookup.
Serialization. Most caches store bytes, not objects. Serialization is turning an object in your program's memory into a sequence of bytes that can be stored or sent, and deserialization is turning it back. This is not free. For small, very hot values, the time spent serializing can be a real fraction of the time you saved — which is one reason keeping a cache inside your own process (Lesson 5) is so fast: no serialization at all.
TTL stands for time to live. It is the number of seconds a cached copy is considered usable. When the TTL runs out, the copy has expired, and the next request for it is treated as a miss.
key: product:42:price:currency=USD value: 25.00 TTL: 300 ← usable for 5 minutes, then thrown away
TTL is your main tool for controlling staleness. A 300-second TTL is you saying: “I accept that this price may be up to five minutes out of date.”
That is a business decision as much as a technical one. Five minutes of a stale product description is fine. Five minutes of a stale stock count might mean selling something you do not have.
TTL is introduced here because you need it from Lesson 3 onward. The policies for what to throw away when memory runs out are Lesson 10 — a related but genuinely different topic.
Cache warming is loading data into a cache before anyone asks for it, so the first real user gets a hit instead of a miss.
Three common approaches:
Warm on deploy. After releasing new code, run a script that requests your most important pages so their entries exist before traffic arrives.
Warm from a log. Take the top few thousand keys from yesterday's traffic and load those. Simple and effective, because yesterday's popular items are usually today's.
Lazy warming. Do nothing special — let the cache fill as real requests arrive. This is the default. It is fine for small caches and dangerous for large ones, because the whole warm-up period runs with your database unprotected.
Warming matters most when your cache is large and your database is only just able to cope without it. If your database can handle full traffic with a cold cache, warming is a nicety. If it cannot, warming is the difference between a deploy and an outage.
“The cache key is just the record's ID.”
The ID is the beginning of the key. The rest is language, currency, permissions, device, feature flags — every input that changes the answer. The forgotten one is where your bug lives.
In a room far from the light, the Latency Lord watches a single request pass by. It asks for product 999999999 — a product that does not exist and never has.
He watches it slip past the browser, which has no copy of it. Past the edge, which has no copy of it. Past the application cache, which has no copy of it. And he watches it strike the database directly, and the database dutifully search its entire index for something that was never there.
He has found the weakness: nonexistent products are not cached, so every new fake ID forces another useless database lookup. He begins generating thousands of fake product IDs.