Caching: The Two Hard Problems, Live
The fastest code is code that doesn't run — TTLs, invalidation, stampedes, and the staleness bargain you're always making.
🎯 By the end you'll understand
How cache layers stack from browser to database and what each layer actually buys you. The three cache population strategies — TTL, event-driven invalidation, and write-through — their costs and when each is appropriate. Why caching is always a staleness bargain even when you don't say so. How a thundering herd materializes when a hot key expires under peak traffic, and two practical mitigations: jitter and request coalescing. The staleness bug that silently corrupts your dashboard for weeks. And how to apply all of this to TraceLab's slow dashboard queries.
KEY INSIGHTEvery cache is a staleness bargain you're making whether you admit it or not
Phil Karlton's famous line — "there are only two hard things in computer science: cache invalidation and naming things" — is funny because it's true and terrible because most engineers only hear the joke, not the warning.
A cache is a lie you tell your users. A true, fast, convenient lie — "here is the answer from last time, which was recent enough that you won't notice" — but a lie. The speed comes from not doing the real work. The risk is that the lie becomes stale and you don't notice until weeks later when a robot that was renamed "Café-3" still shows up as "Café-2" in every dashboard query, because you cached the display name with a 24-hour TTL and never invalidated it on update. The engineers who use caches well understand the bargain explicitly. The ones who get burned by it thought they were just "making it faster."
Every caching decision has three axes: freshness (how stale can the data be before it causes a problem?), hit rate (how often does the cache actually spare the origin work?), and eviction behavior under load (what happens when a hot key expires while a thousand requests are simultaneously waiting for it?). TTLs and invalidation are answers to the first two. Jitter and coalescing are answers to the third — and most teams don't think about the third until it drops production.
HOW IT WORKSCache layers: client → CDN → app → database
Caching is not one thing. Every web request passes through a stack of caches, each with different ownership, different TTLs, and different invalidation reach. Understanding the stack matters because a bug in one layer can make another layer's fix invisible.
Browser HTTP cache
Controlled by Cache-Control response headers: max-age=300 tells the browser to serve the response from disk for 5 minutes without hitting the network. ETag and Last-Modified enable conditional GETs — the browser asks "is this still fresh?" and only downloads the body if it isn't. You don't code this layer; you configure it with headers. Invalidation reach: zero — you cannot push invalidation into a user's browser cache. You can only set a short TTL or use cache-busting URLs (e.g. /app.abc123.js).
CDN edge cache
A distributed network of proxy servers (Cloudflare, Fastly, AWS CloudFront) that cache responses at dozens of geographic locations. A user in Jakarta gets the response from a Singapore PoP, not your origin in us-east-1. CDNs honor the same Cache-Control headers but add their own invalidation API — you can explicitly purge a URL from all edge nodes. This layer is immensely powerful for static assets and reasonably powerful for API responses when you get the cache-key design right (Vary headers, URL structure). TraceLab's dashboard API lives here when you put it behind a CDN — but you need to invalidate on every write, or your dashboard shows stale data to every user worldwide.
Application cache (Redis / Memcached)
Your code explicitly reads from and writes to this layer. Redis is the industry default: sub-millisecond reads, rich data structures, TTL-per-key, pub/sub for invalidation signals, and enough production history to trust. The application cache is where most of this lesson lives — it's the layer you own completely, where TTL vs. invalidation decisions are made in code you write, and where thundering herds happen when you don't add jitter.
Database query cache
PostgreSQL's shared_buffers keep frequently-accessed data pages in memory; the query plan cache stores parsed execution plans. These are automatic, opaque to your application code, and invaluable — a query that runs in 200ms the first time may run in 8ms when the data is already in buffer. This cache buys you less than you think once your working set exceeds shared_buffers, because the buffer eviction algorithm then controls your hit rate, not you. The practical implication: slow queries in production that were fast in your dev environment often mean a cold buffer cache, not a bad query plan.
CORE CONCEPTTTL, event-driven invalidation, and write-through
You have three strategies for deciding when cached data is replaced. Each is the right answer in specific situations and the wrong answer in others — the most common mistake is applying TTL where event-driven invalidation is required.
| Strategy | How it works | Staleness window | Best when | Worst when |
|---|---|---|---|---|
| TTL (time-to-live) | Cached entry expires after N seconds; next read causes a cache miss and re-fetches | Up to N seconds — predictable but non-zero | Data changes on a known schedule, or occasional staleness is acceptable (e.g. analytics aggregations) | Data changes on user action (profile edits, robot renames); you care about exact consistency |
| Event-driven invalidation | On write, explicitly delete or update the cached key; next read populates from origin | Near-zero — stale only during the race between write and invalidation | Writes are infrequent, well-defined, and you control the write path | Multiple services can write the same data; you can't enumerate all cache keys to invalidate; write volume is high |
| Write-through | Every write updates both origin and cache in the same operation; reads always hit cache | Zero — cache is always current | Read-heavy data that changes on a known path; latency budget allows the double write | Cache nodes fail and you need to backfill; writes need to be atomic with cache update (complex); cache is much smaller than origin dataset |
Event-driven invalidation sounds simple: on write, delete the key. The hidden cost is that real systems have derived keys. If you cache dashboard:summary:robot_id:42, dashboard:leaderboard:all, and dashboard:summary:all_robots — and a write to robot 42 should invalidate all three — your invalidation logic needs to know about all of them. When that mapping lives in the same code as the write, it's manageable. When five different services can write to the same underlying data, you've just invented a distributed cache coherence problem.
There is a fourth, often-overlooked pattern: stale-while-revalidate. The cache serves the stale value immediately and simultaneously fires an async request to refresh it in the background. The user gets fast (stale) data; on the next request the fresh data is ready. HTTP has this semantics built into Cache-Control: stale-while-revalidate=60. Redis doesn't, so you implement it yourself: serve the expired key, schedule an async refresh, set a short "being refreshed" flag to prevent duplicate refreshes. The staleness window is bounded by how long the background refresh takes, not by the TTL itself.
SEE ITCache strategy simulator
TraceLab's dashboard runs a slow aggregation query — it scans the last 10,000 runs, computes per-robot statistics, and returns a leaderboard. At idle the query takes 80ms; at peak traffic (50 concurrent users) it climbs to 600ms. Put a cache in front of it and tune the parameters to understand the tradeoffs. Then press "Expire hot key under load" to trigger the thundering herd — and fix it.
Two things to internalize from the simulator. First, notice that even with write-through — the "zero staleness" strategy — the herd burst drops to 1 (coalescing works) but origin load at peak is still non-trivial because write-through forces an origin write on every cache entry update. The cost trades positions; it doesn't vanish. Second: with TTL-only and no mitigation, expiring the hot key under 100 concurrent users produces a burst of 100 simultaneous origin queries — each one runs the 600ms aggregation. That's 60 seconds of cumulative database work arriving in under a second. With jitter, those expirations spread over a window; with coalescing, only one query runs and the rest wait for its result.
WEAKNESSThe thundering herd: how a cache failure causes an outage
The thundering herd (also called a cache stampede) is one of caching's most reliable failure modes. The sequence: a hot cached key expires. Simultaneously, N requests — all of which would have been cache hits — arrive to find the key missing. All N treat this as a cache miss and fire a query to the origin simultaneously. The origin, which was calmly handling 1–2 queries per TTL interval, now receives N queries in the same instant. If N × query_cost exceeds the origin's capacity, it falls over. The cache is now empty, the origin is down, every request is a miss, and you have a cascading outage where your performance optimization is the root cause.
Reddit's engineering team documented cache stampede cascades multiple times in their early scaling era. A popular post would expire its cached comment count; the next request wave would all regenerate it simultaneously; the database would buckle; the site would go down. The fix was a combination of probabilistic early expiration and dog-pile locks — now called "jitter" and "request coalescing" in the industry. The pattern is old enough to have a Wikipedia entry and young enough that your team will independently rediscover it if you cache a hot key under load without mitigations.
Mitigation 1 — TTL jitter. Instead of a fixed TTL, set TTL = base_ttl + random.randint(-jitter, +jitter). Keys that were populated together now expire at different times, spreading the miss events across a window instead of synchronizing them. The staleness window widens by the jitter range — that's the explicit cost. In TraceLab, a base TTL of 30 seconds with ±10 second jitter means the dashboard leaderboard cache entries expire between 20 and 40 seconds, so at most a few requests regenerate it simultaneously rather than hundreds.
import random BASE_TTL = 30 # seconds JITTER = 10 # ± seconds def cache_set(redis_client, key, value): ttl = BASE_TTL + random.randint(-JITTER, JITTER) redis_client.setex(key, ttl, value) # key expires at a unique time
Mitigation 2 — Request coalescing (probabilistic lock). When a cache miss occurs, the first request acquires a short-lived lock (typically another Redis key) and performs the origin query. Subsequent requests that arrive during the same miss window find the lock set; they wait briefly (or serve the stale value if available) rather than firing their own origin query. When the first request completes and populates the cache, it releases the lock; all waiting requests then read the freshly populated key.
import time LOCK_TTL = 5 # seconds; must exceed worst-case origin latency def get_with_coalesce(redis_client, key, compute_fn): val = redis_client.get(key) if val is not None: return val # fast path: cache hit lock_key = key + ":lock" if redis_client.set(lock_key, "1", nx=True, ex=LOCK_TTL): # we won the race — compute and populate try: result = compute_fn() cache_set(redis_client, key, result) return result finally: redis_client.delete(lock_key) # lost the race — poll briefly for the winner's result for _ in range(20): time.sleep(0.1) val = redis_client.get(key) if val is not None: return val return compute_fn() # fallback if winner timed out
A longer TTL reduces the frequency of expiration events, which reduces herd probability — but at the cost of wider staleness windows. If your data changes every 30 seconds and your TTL is 10 minutes, you're serving stale data for 9 minutes and 30 seconds after every write, 95% of the time. The herd mitigation is better because it keeps the staleness window short while eliminating the stampede risk. You want both: jitter to spread expirations, coalescing to handle the ones that still collide.
EXAMPLEThe staleness bug that takes weeks to notice
The other classical caching failure is slower and more insidious: a cached key is invalidated correctly on the path you knew about but not on the path you forgot. The data drifts silently. Users start seeing inconsistencies they can't reproduce because the inconsistency depends on which cache node served their request, and whether the cache was warm when they asked.
Here's the TraceLab version. You cache the dashboard leaderboard with a 5-minute TTL and add event-driven invalidation that fires whenever log_run() is called. Looks solid. But then someone calls rename_robot() — a new function added in a hotfix — which writes to the database but doesn't fire the invalidation event because nobody updated the cache layer. The leaderboard now shows the old robot name for up to 5 minutes. Fine. But 3 weeks later someone extends the TTL to 24 hours for a specific analytics endpoint that is genuinely slow, and now the stale name persists for a day.
The failure is not a bug in the caching logic — it's a missing connection between a write path and the invalidation logic. The pattern that prevents it:
Enumerate all cache keys that a given write affects
In TraceLab's code, keep a mapping: CACHE_KEYS_FOR_WRITE = { 'log_run': ['dashboard:leaderboard:*', 'run:detail:{id}'], 'rename_robot': ['dashboard:leaderboard:*', 'robot:detail:{id}'] }. Every write function looks up its entry and invalidates all listed keys. If you add a new write function without updating this map, a test fails.
Test staleness explicitly
Write a test that performs a write, immediately reads the cached value, and asserts it reflects the write. This is not a unit test — it requires a real (or fake) cache. Add it to your integration test suite. A failing test here means "a write path is not invalidating correctly," which is far better than a failing user report 3 weeks later.
Set a maximum TTL across all keys
Even with event-driven invalidation, set a fallback TTL. If invalidation fails silently (network partition, bug), the TTL is the backstop. Set it to the longest staleness your users would accept and document why. If you can't accept any staleness, you shouldn't be caching — or you need write-through with a very reliable invalidation path.
TRACELABCaching the TraceLab dashboard queries
TraceLab v2 has a dashboard that runs three expensive queries on every page load: a leaderboard of robot runs sorted by reward (scans all runs), a time-series of loss values per experiment (scans metrics), and an aggregate summary (joins runs × metrics × robot names). Under load from 20 concurrent users, the database connection pool exhausts and the dashboard times out. The fix is a cache in front of all three queries — but which strategy?
import json, redis, time, random r = redis.Redis(host="localhost", port=6379, decode_responses=True) LEADERBOARD_KEY = "tracelab:dashboard:leaderboard" LEADERBOARD_TTL = 30 # seconds — staleness users won't notice JITTER = 10 # ± seconds to spread expirations def get_leaderboard(db) -> list: # 1. try cache first cached = r.get(LEADERBOARD_KEY) if cached: return json.loads(cached) # cache hit: ~1 ms # 2. cache miss — acquire coalescing lock lock_key = LEADERBOARD_KEY + ":lock" if r.set(lock_key, "1", nx=True, ex=5): try: rows = db.query_leaderboard() # slow query: ~80 ms ttl = LEADERBOARD_TTL + random.randint(-JITTER, JITTER) r.setex(LEADERBOARD_KEY, ttl, json.dumps(rows)) return rows finally: r.delete(lock_key) # 3. lost the lock race — wait for winner's result for _ in range(20): time.sleep(0.1) cached = r.get(LEADERBOARD_KEY) if cached: return json.loads(cached) return db.query_leaderboard() # fallback def invalidate_leaderboard(): # called by log_run(), rename_robot(), delete_run() on every write r.delete(LEADERBOARD_KEY) # next read re-populates
The design choices here are explicit. TTL is 30 ± 10 seconds — fast enough that a new run appears on the leaderboard within 40 seconds at worst, and short enough that a missed invalidation event (say, a background job writing directly to the database) heals itself quickly. Jitter prevents coordinated expiration when all 20 dashboard tabs load on the hour. Coalescing prevents the stampede on the first miss after startup or a cache flush. And invalidate_leaderboard() is a one-liner that all write paths call — not a complex pub/sub system, because TraceLab v2 is still a monolith and all writes happen in the same process.
The above pattern is cache-aside (also called lazy loading): the application checks the cache, handles misses, and populates the cache itself. It's the right default because it gives you full control, works with any cache store, and doesn't require the cache to know about your data model. Write-through and read-through move this logic into a cache client library — your code just reads and writes normally, and the library handles the cache layer transparently. These patterns are appealing but create tight coupling between your data access layer and your caching strategy, making invalidation bugs harder to trace. Start with cache-aside; reach for read/write-through only if the application-layer bookkeeping becomes unmanageable.
IN PRACTICEWhen NOT to cache — the honest verdict
Caching is a tool, not a virtue. Several situations where reaching for a cache is the wrong answer:
Adding a cache to a fast query is waste with a staleness budget attached. A 2ms query doesn't need a cache — it needs you to stop adding a cache. Profile first. TraceLab's leaderboard query at 80ms under load is a real candidate. A simple metadata fetch at 3ms is not. The rule: if the uncached path is acceptable, the cache is premature complexity.
Financial systems, safety-critical sensor readings, experiment run status on a robotics platform — if users need to know whether the robot is running right now, a 30-second cached value is actively dangerous. Cache the slow, tolerant parts (aggregated leaderboards, historical statistics); don't cache the real-time parts (current robot state, live metric stream). These have different freshness requirements and should be separate data paths.
A cache that is constantly being invalidated isn't caching anything — it's an expensive lookup system with a bug. If your hit rate in the simulator won't climb above 10% even with a generous TTL and moderate concurrency, the access pattern is write-heavy or the keys are too unique. Address the database query directly (indexes, denormalization, precomputed aggregates) rather than papering over it with a cache. TraceLab's per-run detail endpoint is like this: every run has a unique ID, so caching individual run detail pages only helps if the same run is fetched repeatedly, which it isn't.
WEAKNESSWhere caching bites
Cached permission checks are a security vulnerability waiting to happen. A user is removed from a project at 09:00; the permission cache expires at 09:05; they make five API calls in those five minutes that succeed and shouldn't have. TraceLab's access control must not go through the cache. This sounds obvious until you're optimizing a slow authorization check at 3 AM and the cache is right there. Never cache deny/allow decisions unless you have explicit, tested, zero-latency invalidation on every role or permission change.
Two different users querying dashboard:leaderboard should see the same data — that key is shared. But if the leaderboard is personalized (user A sees only their robots, user B sees all), the cache key must include the user context: dashboard:leaderboard:user:42. Missing this gives user B's data to user A. This is not a theoretical mistake. The production incident playbook for every major platform includes at least one "we cached a response with user data using a key that didn't include the user ID." Always include every dimension that affects the query result in the cache key.
Redis goes down. The node restarts. The memory limit is hit and keys are evicted. Your cache layer is a best-effort performance optimization, not a data store. Your application must handle every cache miss — including mass miss events — without breaking. If your code path has if cache is None: raise ValueError anywhere in the query path, you've taken a soft dependency and made it hard. The fallback to the origin query must be tested, not theoretical.
KEY INSIGHTHow this connects
- Boundaries (4.1) — cache design is a boundary decision: you're adding a layer with its own interface, failure modes, and consistency contract. A cache that spans a service boundary multiplies the invalidation problem.
- Architecture styles (4.2) — the thundering herd is a distributed-systems failure mode: it only matters at a scale where you have enough concurrent users to stampede. A single-user TraceLab doesn't need request coalescing.
- Queues and async work (4.4) — stale-while-revalidate is a queue in disguise: you enqueue a background refresh and serve the stale value while it processes. The next lesson makes this pattern explicit and general.
- Databases II: Queries, Indexes, and the Planner (3.5) — the slow query that caching papers over is often a missing index or a bad query plan; before reaching for Redis, run
EXPLAINand fix the query. Caching and indexes are not alternatives — the index reduces the cost of the query the cache can't serve, namely the one that involves fresh data. The N+1 query pattern that 3.5 demonstrates is exactly the access pattern that drives hit rates to zero and makes caching useless.
Every architectural component added in Stage 4 carries an explicit cost and a specific problem it solves. Caching solves read latency at the cost of staleness, complexity, and new failure modes. The lesson's through-line — you're always making a staleness bargain whether you name it or not — applies at every layer, from Cache-Control headers to Redis TTLs to PostgreSQL's shared_buffers. Name the bargain, set it deliberately, and test what happens when the cache is empty.
GLOSSARYKey terms introduced
- TTL (time-to-live)
- A duration after which a cached entry is considered expired and will be re-fetched from the origin on the next request. Simple to implement; guarantees a bounded staleness window.
- Cache invalidation
- Explicitly removing or updating a cached entry when the underlying data changes, rather than waiting for TTL expiry. Gives near-zero staleness but requires all write paths to participate.
- Write-through caching
- A strategy in which every write updates both the origin store and the cache in the same operation. The cache is always current; read latency is always low. Adds write latency and cache-warming complexity.
- Thundering herd (cache stampede)
- When a popular cache key expires and many concurrent requests simultaneously treat it as a cache miss, all firing origin queries at once. Can overwhelm the origin and cause a cascading outage.
- TTL jitter
- Adding a random offset to each key's TTL so that keys populated at the same time expire at different times, spreading out the miss events and reducing stampede probability.
- Request coalescing (dog-pile lock)
- A mitigation for the thundering herd: when a cache miss occurs, one request acquires a short-lived lock and performs the origin query; all other requests wait for its result rather than firing their own queries.
- Cache-aside (lazy loading)
- The most common application cache pattern: the application checks the cache, handles misses explicitly, and populates the cache after an origin fetch. The application owns all cache logic.
- Stale-while-revalidate
- A pattern that serves a stale cached value immediately and simultaneously fires an async refresh. Balances freshness and latency by making the refresh asynchronous relative to the user's request.
ACTIVE RECALLCheck your understanding
Answer in your own words first, then reveal.
1A thundering herd happens when a hot key expires. What causes the herd, and what are the two standard mitigations?›
The herd is caused by N concurrent requests all arriving to find the same cache miss simultaneously, all firing origin queries instead of exactly one. The two mitigations: TTL jitter spreads expiration times so keys populated together don't expire together, reducing the probability of synchronized misses; request coalescing (dog-pile lock) ensures that when a miss does occur, only one request executes the origin query and all others wait for its result.
2What is the staleness bargain, and why is it always present even with write-through caching?›
The staleness bargain is the tradeoff you make when you cache: you accept that some reads may return data that doesn't reflect the latest writes, in exchange for lower latency. With TTL it's explicit — up to N seconds stale. With event-driven invalidation it's near-zero but non-zero (there's a race between the write and the invalidation). Write-through is near-zero for the writes it processes, but still has a window between write and cache update during write-path failures. More subtly, write-through caches are often partial — not every key gets a write-through update — so some keys still rely on TTL as a backstop. The bargain is always there; write-through just makes it very small.
3You add event-driven invalidation to TraceLab's leaderboard cache — invalidate_leaderboard() is called from log_run(). Six months later the leaderboard shows stale robot names. What went wrong?›
A write path was added — rename_robot() — that updates the robot name in the database but doesn't call invalidate_leaderboard(). The cache key depended on robot names, but the invalidation logic wasn't updated when a new name-changing write path was introduced. The pattern that prevents this: maintain a mapping of "which write functions affect which cache keys" and test that every write path invalidates all relevant keys. A missing entry in the mapping is a bug; the test makes it visible before production.
4Name two situations where adding a cache is the wrong answer, and explain why.›
1 — Authorization decisions: caching permission checks introduces a window where a revoked user can still access resources. The staleness window (however short) becomes a security vulnerability, not just a UX annoyance. 2 — High-write, unique-key access patterns: if every write invalidates the cached key and every read is for a different key, the hit rate is near zero. You're paying for a cache layer with no benefit. The real fix is usually a database index or a denormalized precomputed aggregate, not a cache. Other valid answers: caching without measuring the uncached path first (premature complexity); caching real-time safety-critical data with a non-zero staleness window.