Concurrency — Engineering Journal

Jul 02, 20269 min

The ABA Problem: When Compare-And-Swap Said Nothing Changed and It Lied

A lock-free stack built on compare-and-swap passed every load test and then corrupted itself in production, popping a node that had already been freed. The CAS never failed, and a failing CAS was the only case anyone had thought to worry about. The bug was a value swinging from A to B and back to A while a paused thread was not looking, so the compare saw the same bit pattern and happily proceeded on data that was no longer the data it started with.

ConcurrencyLock-Free

Read

Concurrency

Jun 28, 20269 min

The Lock That Held Two Owners: A GC Pause Versus a Redis TTL

We used a Redis lock with a TTL to make sure exactly one worker processed each payout. For months it worked, then a batch ran twice and double-charged a handful of accounts. The lock was doing its job perfectly. The problem was that a stop-the-world GC pause outlived the lock's expiry, so a worker that believed it still held the lock kept working while a second worker had already taken it.

ConcurrencyDistributed Systems

Read

Concurrency

Jun 24, 20268 min

The Connection Pool Deadlock: When One Request Needs Two Connections

The endpoint was fine in every test and fine in staging. Then traffic crossed some invisible line in production and it stopped responding entirely, not slow, fully frozen, with no errors in the logs. The pool had ten connections, and under load every request was holding one while waiting for a second that would never come.

ConcurrencyDatabases

Read

Concurrency

Jun 05, 20258 min