The Last CachebenderInterlude · Spirit
Interlude · Spirit
Interlude · Spirit · Lesson 10 of 14
Lesson Ten

The Temple of Eviction

Technique unlocked The Letting Go expiry and eviction

Two different reasons data leaves

Things disappear from a cache for two reasons, and the two have nothing to do with each other.

  1. Expiration — the copy got old, so you stopped trusting it.
  2. Eviction — you ran out of room, so something had to go.

That is the whole distinction. It is worth stating that plainly, because the two are confused constantly — including in documentation.

Trigger Reason Controlled by
Expiration Time The copy is too old to trust TTL
Eviction Memory pressure You ran out of room Eviction policy

Expiration is time-driven. It is about correctness. The value may still be perfectly useful and there may be plenty of memory — you remove it because you no longer believe it.

Eviction is memory-driven. It is about capacity. The value might be perfectly current and useful — you remove it because something else needs the space.

A cache with infinite memory still needs expiration. A cache holding data that never changes still needs eviction. They are independent, and confusing them leads to setting TTLs to manage memory (which makes your data needlessly stale) or sizing memory to manage freshness (which does not work at all).

TTL, used deliberately

You met TTL in Lesson 2. Now use it for two distinct jobs.

TTL as a correctness tool. The TTL is your statement about how wrong you are willing to be. TTL = 300 says: I accept this may be up to five minutes out of date. Pick it from the business requirement, not from habit.

TTL as a safety net. This one is subtle and important. Even with perfect invalidation (Lesson 11), things go wrong — a message is dropped, a service is down, a code path is missed. TTL bounds the damage. Without it, a failed invalidation means a value is wrong forever. With it, the worst case is bounded by the TTL.

Put a TTL on everything, even data you invalidate explicitly. It is the backstop for the day your invalidation silently fails. It will.

TTL jitter

Set 10,000 keys during a deployment, all with TTL = 3600.

One hour later, all 10,000 expire in the same second.

Jitter is adding a small random amount to each TTL so they spread out:

ttl = 3600 + random(0, 600)     # between 60 and 70 minutes

Now those 10,000 keys expire spread across ten minutes rather than in a single instant. This single line prevents a specific attack in Lesson 12, and it costs you nothing.

Eviction policies

When the cache is full and something new arrives, something old must go. The eviction policy decides what.

Reference — expand as needed
LRU — Least Recently Usedoptional read

Remove whatever has not been touched for the longest time.

The idea is simple: recently used items are often used again soon. That makes LRU a useful starting point.

The weakness — and it is severe. LRU is destroyed by a scan: one large pass over data that will not be requested again. A nightly report reading a million products marks all of them “recently used,” pushing out the small hot set that your actual users depend on. The report finishes; your cache is full of things nobody wants; your hit ratio collapses; your database absorbs the difference.

A single batch job can wipe out a perfectly healthy cache. Watch for this.

Where you would use it Almost everywhere, as the default. It is what Redis allkeys-lru gives you, what most application cache libraries do without being asked, and what your CPU does with its own caches. Session data fits it especially well — a user who was active a minute ago is the one most likely to be back. Reach for something else only if you have batch jobs sweeping the cache, or a hot set stable enough for LFU to protect.

LFU — Least Frequently Usedoptional read

Remove whatever has been requested least often, regardless of when.

The logic: popularity is more meaningful than recency. A page requested 10,000 times matters more than one requested once, even if the once was more recent.

Why it can beat LRU: it survives scans. The report's million products each get one access; the front-page item has thousands. LFU keeps the front page and discards the report data — exactly right.

The weaknesses:

  • New items look unpopular. Something added a second ago has a count of 1 and is a prime eviction target, even if it is about to become the most popular item on the site. New content struggles to establish itself.
  • Old items overstay. Something enormously popular last month keeps a high count long after people stopped wanting it.

Practical implementations use windowed or decaying LFU, counting recent accesses more heavily, which fixes both. If you choose LFU, choose one of these.

Where you would use it When a small, stable set of items takes most of the traffic — the top 1% of products that account for most views, a CDN holding assets that stay popular for months, reference data that barely changes. It is also the right answer when you have a nightly batch job that would otherwise flush an LRU cache. Avoid it where content turns over quickly: news, social feeds, anything where today's popular item was created this morning.

FIFO — First In, First Outoptional read

Remove whatever was added earliest, regardless of use.

Simple and cheap — no bookkeeping. It also ignores usage entirely, so it will happily evict your most popular item because it happened to be cached first. Rarely the right choice for a cache; fine for a queue.

Where you would use it Honestly, rarely, and almost never for a general cache. Where it does fit is anywhere insertion order is the point rather than an accident — a fixed-size buffer of recent events, a replay window, a rolling log kept in memory. If every entry has roughly equal value and you want zero bookkeeping, FIFO is defensible. If some entries are more valuable than others, it is the wrong tool.

Randomoptional read

Remove a randomly chosen item.

This sounds like giving up, and it is not. It requires no bookkeeping at all, which makes it very fast and memory-efficient — LRU must maintain ordering on every single access, and that costs something on every read.

For workloads without a strong recency pattern, random performs surprisingly close to LRU at a fraction of the overhead. Redis offers it deliberately. Do not dismiss it.

Where you would use it Very high-throughput caches where LRU's per-read bookkeeping is a measurable cost, and memory-constrained systems where the ordering structure itself is too expensive to keep. It also does well where access is genuinely uniform and no hot set exists to protect. Redis exposes it as allkeys-random for exactly these cases.

Choosing

Policy Best when Fails when
LRU Access is bursty and recency-driven A scan sweeps through it
LFU There is a stable hot set Content turns over quickly
FIFO Almost never for caches Any time popularity matters
Random Low overhead matters; access is uniform A clear hot set deserves protecting

Real caches offer combinations — Redis has allkeys-lru, volatile-lru (only keys with a TTL set), allkeys-lfu, volatile-ttl (shortest remaining life first), and noeviction (reject writes when full).

volatile-* variants matter more than they look: they let you mark some entries as never-evictable by giving them no TTL, protecting data you cannot afford to lose.

Reading the eviction rate

The eviction rate — how many items are being removed per second for lack of space — is one of the most useful signals a cache gives you.

  • Near zero: your cache is big enough. Good.
  • High and rising: your cache is too small. Items are being thrown out before they can be reused, and your hit ratio is being destroyed by capacity, not by TTL.
  • Sudden spike: something just flooded the cache. A batch job, a scan, a bug writing far too much.

A high eviction rate paired with a falling hit ratio is a specific, actionable diagnosis: the cache needs more memory, or you are caching too much. Lesson 13 will show you where to watch this.

At the Storefront

Part TTL Reasoning
Product details 1 hour + jitter Changes rarely; an hour stale is harmless
Price 60 seconds Must be close to correct
Stock count 10 seconds, or uncached Being wrong here sells things you do not have
Reviews 10 minutes + jitter Slow-changing, heavily read
Category tree 5 minutes Small, hot, rarely edited

Policy for the Redis instance: allkeys-lru as the default — and check for batch jobs that scan products, because those will wreck it. If you have one, either route it around the cache or move to windowed LFU.

Common misconception

“LRU is always the right choice.”

LRU is a good default that a single batch job can destroy. LFU survives scans but starves new content. Random is cheaper than both and often nearly as good. Choose from your access pattern, not from habit.

The Forge

The Latency Lord walks the realm at a quiet hour and does something very simple.

He does not delete anything. He does not break anything. He simply finds every expiry timer in the kingdom and sets them all to the same hour.

Nothing is stolen. Nothing is broken. Everything simply ends at once.