1. What is the Thundering Herd Problem?
In computing, the "thundering herd" describes a situation where a large number of processes or threads are woken up simultaneously to respond to a single event, but only one of them can actually do the work. The rest waste resources and contend with each other before giving up.
In caching systems, the most common manifestation is the cache stampede: a popular cached item expires, and all concurrent requests for that item experience a cache miss at the same time. Without coordination, every one of those requests goes to the database to fetch the data — resulting in a sudden spike of identical queries that can overwhelm the DB.
NORMAL (cache warm):
1,000 req/s for "product:best-sellers"
ALL hit Redis → sub-ms response, DB receives 0 queries
THUNDERING HERD (key expires at T=0):
Request #1: cache MISS → query DB (in-flight)
Request #2: cache MISS → query DB (in-flight)
Request #3: cache MISS → query DB (in-flight)
... 1,000 requests/sec, each spawning identical DB query
Result: DB receives 1,000 simultaneous identical queries
DB CPU spikes to 100% → cascading failure
2. Solution 1 — Cache Lock (Mutex)
The simplest and most reliable fix is to use a distributed lock (mutex) to ensure only one request fetches the data from the database when a cache miss occurs. All other requests wait for the lock holder to finish and then read the freshly cached value.
Serve Stale While Revalidating
A better variant: store the data with both a "soft TTL" (when to refresh) and a "hard TTL" (when to discard). On soft-TTL expiry, immediately serve the stale value and trigger an async background refresh. Only when the hard TTL hits do you block and wait. This is the stale-while-revalidate pattern used by CDNs and service workers.
3. Solution 2 — Probabilistic Early Expiry (XFetch)
Probabilistic early expiry triggers a cache recompute before the key expires, with increasing probability as the TTL counts down. As expiry nears, the algorithm rolls the dice more aggressively, ensuring the cache is refreshed before any request ever sees a miss.
The XFetch algorithm: recompute if current_time - delta * beta * ln(random()) > expiry_time, where delta is the time the last recompute took (in seconds) and beta is a tuning constant (typically 1.0). Higher beta means earlier and more aggressive refresh.
4. Solution 3 — Staggered TTLs (Jitter)
If you cache the same data in multiple shards or replicas with identical TTLs, all copies expire simultaneously. Adding random jitter to the TTL staggers expiries over time:
5. Solution 4 — Background Refresh
Instead of waiting for a key to expire before refreshing it, a background job proactively refreshes hot keys before they expire. This approach requires knowing which keys are "hot" — typically tracked by monitoring access frequency (e.g. Redis key access counters or application-level frequency tracking).
- A background worker scans keys with TTL below a threshold (e.g. TTL < 60 seconds)
- It re-fetches the data from DB and extends the TTL
- No request ever experiences a cache miss for hot keys
- Works well for a small, known set of hot keys (e.g. homepage data, trending items)
6. Solution 5 — Request Coalescing
Request coalescing ensures that when multiple requests arrive for the same cache-miss key, only one upstream DB request is made. The remaining requests wait for the first to complete and are then all served the result.
This is implemented at the reverse-proxy or cache layer:
- Nginx proxy_cache_lock: when multiple requests arrive for the same uncached resource, only the first is forwarded upstream. Others wait (
proxy_cache_lock_timeout) and then serve from cache. - Varnish grace mode: serves stale objects while one request fetches fresh content from the origin.
- CDN request collapsing: Fastly, Cloudflare, and Akamai all implement request collapsing at the edge — a feature called "request coalescing" or "request collapsing."
7. Thundering Herd in the Linux Kernel
The thundering herd problem is not limited to caches. In earlier Linux versions, the accept() system call on a shared socket would wake all processes waiting for a new connection — even though only one could accept it. The rest would wake up, fail to acquire the connection, and go back to sleep, wasting CPU on context switches at high load.
Modern Linux fixes:
- SO_REUSEPORT (Linux 3.9+): allows multiple processes to bind to the same port. The kernel distributes incoming connections across the processes, waking only one per connection.
- EPOLLEXCLUSIVE (Linux 4.5+): when multiple processes share an epoll set, EPOLLEXCLUSIVE ensures only one is woken per event, eliminating the thundering herd on I/O events.
8. Comparison of Solutions
| Solution | Complexity | Prevents Stale Reads? | Adds Latency? | Best For |
|---|---|---|---|---|
| Cache Lock (Mutex) | Medium | Yes | Yes (waiters block) | General-purpose, correctness-critical |
| Stale-While-Revalidate | Low | No (brief stale period) | No | Content where brief staleness is OK |
| Probabilistic Early Expiry | Medium | Yes | No | Known expensive-to-compute keys |
| Staggered TTLs (Jitter) | Very Low | Partially | No | Preventing synchronized multi-shard expiry |
| Background Refresh | High | Yes | No | Known hot keys, stable data |
| Request Coalescing | Low (proxy-level) | Partially | Yes (waiters block) | CDN/reverse-proxy layer |
Thundering Herd on Cache Warmup After Restart
A thundering herd also occurs when you restart a cache server (e.g. Redis reboot) and all cached data is gone. Every request becomes a cache miss. The fix: use a cache warming strategy — pre-populate the cache before sending production traffic (blue/green deploy pattern), or use a slow traffic ramp-up after restart. Never restart your primary cache node during peak traffic without a warm replica ready to take over.
How We Research and Update This Guide
We test the underlying formula or workflow, compare outputs with reliable references, and revise examples whenever the page content changes.
- The workflow or formula is tested directly in the tool and compared against independent reference examples.
- Examples are kept practical so readers can verify the result without hidden assumptions.
- Pages are revised whenever the interface, calculation flow, or surrounding guidance materially changes.
Frequently Asked Questions — Thundering Herd Problem
The thundering herd problem occurs when a popular cached item expires and many concurrent requests — all experiencing a cache miss simultaneously — race to fetch the same data from the database. Instead of one DB query, you get hundreds or thousands simultaneously, potentially overwhelming the database. This is also called a cache stampede. It is most severe for hot keys accessed thousands of times per second.
When a cache miss is detected, the first request acquires a distributed lock (mutex) for that cache key before fetching from the database. All other requests that arrive during the fetch either wait for the lock to be released and then read the freshly cached value, or they return a stale value if one is available. The lock prevents duplicate DB queries for the same key. The lock TTL must be longer than the expected DB fetch time to avoid lock expiry during the query.
Probabilistic early expiry (also called XFetch) is a technique where a cache entry is recomputed before it actually expires. As the TTL decreases, the probability of triggering a recompute increases. The formula is: recompute if current_time + beta * delta * ln(random()) > expiry_time, where delta is the time the last recompute took and beta is a tuning parameter (usually 1). This spreads recomputation over time before expiry rather than letting it happen all at once.
Request coalescing (or request collapsing) groups multiple simultaneous requests for the same resource into a single upstream request. When the first request for a cache-miss key arrives, it is registered as "in-flight." Subsequent requests for the same key while the first is pending are held and wait for the first request to complete. When the result arrives, all waiting requests are satisfied simultaneously. This is used in Nginx proxy_cache_lock, Varnish, and CDN edge servers.
If all instances of a hot cached item have the same TTL, they all expire simultaneously. Staggered TTLs add a random jitter to the base TTL so expiries are spread over time. For example, instead of a fixed 300-second TTL, use TTL = 300 + random(0, 30). This ensures that at most one shard expires at any given second rather than all shards at once.
In early Linux kernel versions, the accept() system call would wake ALL processes sleeping on a socket when a new connection arrived, even though only one could accept it. The rest would wake up, find nothing to do, and go back to sleep — wasting CPU on context switches. Linux 3.19+ fixed this with SO_REUSEPORT, which distributes incoming connections across multiple listener sockets, waking only one process per connection event.