Patterns that show up in every senior backend interview and every real outage postmortem — idempotency keys, outbox, sagas, rate limiting, circuit breakers, and the API design rules that keep systems composable.
11 articles · updated regularly
When a hot cache key expires under heavy traffic, thousands of concurrent requests miss at the exact same millisecond and attempt to recompute the same expensive value, crushing the database. We compared distributed locking, Stale-While-Revalidate, and probabilistic early expiration (XFetch) to solve the cache stampede.
A rate limiter rejected traffic smoothly in every test and then let a client through at ten times its contracted rate during a real burst, taking down a downstream dependency that had no defenses of its own. The limiter was not broken. It was a token bucket, correctly implemented, doing exactly what a token bucket is designed to do, which turned out to be the wrong algorithm for a downstream that could not absorb bursts at all.
We moved a service to a new host, repointed the DNS record, watched traffic shift over, and shut the old box down. Then the error rate climbed and stayed up for almost an hour, because a meaningful slice of clients were still sending requests to an IP that no longer answered. Nobody had ever looked at the record's TTL.
The checkout endpoint had a 30-second timeout. So did the payment service it called, and the fraud check that called, and the database underneath that. When one of them got slow, the timeouts stacked instead of capping, and a single sluggish dependency held thousands of connections open until the whole chain fell over.
A downstream service hiccupped for two seconds. Every caller retried three times. Those retries tripled the load on a service that was already struggling, and the blip became a 20-minute outage. Backoff without jitter made it worse, not better.
HTTP/2 multiplexes streams over one connection, so people assume one slow response can't block the others. It can, just one layer down, in TCP. Here is the head-of-line blocking that survived HTTP/2 and why HTTP/3 had to abandon TCP entirely to kill it.
Connection exhaustion under load almost never means you ran out of memory — it means you ran out of ephemeral ports to TIME_WAIT sockets. Here is the TCP lifecycle that explains it and the fixes that actually work.
Every public API needs a rate limiter, but the algorithm you pick changes the burst behaviour your users feel. A practical comparison with Redis implementations.
When a business operation spans several services, you can't wrap it in one ACID transaction. Sagas trade atomicity for compensations — here's how to get them right.
You can't atomically commit to Postgres and Kafka. The outbox pattern is how senior backends finesse that without losing events.
Networks fail, clients retry, and one charge becomes two. Idempotency keys turn that from a postmortem into a non-event.