Clear, practical essays on how real software gets designed, breaks, and ships. The failure modes I've run into, the trade-offs that turned out to matter, and the patterns that hold up in production.
When repl-backlog-size is undersized, brief network hiccups force replicas into full SYNC. How COW memory inflation and diskless replication prevent cascading primary stalls.
Enabling unclean.leader.election guarantees partition uptime at the expense of silent log truncation and message drops. How ISR fencing and min.insync.replicas protect financial state.
PostgreSQL's INSERT ... ON CONFLICT DO UPDATE is commonly treated as an atomic, concurrency-safe upsert. Under high concurrent write load with multi-row batches or multiple unique indexes, it can trigger silent 40P01 deadlocks. Here is how Postgres locks index tuples and how to eliminate the deadlock.
When building high-performance network servers with Linux epoll, Edge-Triggered (EPOLLET) mode promises fewer syscalls than Level-Triggered mode. But without careful loop management, a single greedy socket streaming large payloads can starve all other file descriptors in the event loop.
When a hot cache key expires under heavy traffic, thousands of concurrent requests miss at the exact same millisecond and attempt to recompute the same expensive value, crushing the database. We compared distributed locking, Stale-While-Revalidate, and probabilistic early expiration (XFetch) to solve the cache stampede.
Storage engines must guarantee that a sudden power loss or database crash mid-transaction leaves the data file completely uncorrupted. We compare Shadow Paging (used in LMDB and SQLite journal mode) against Write-Ahead Logging (WAL, used in PostgreSQL, InnoDB, and RocksDB).
A Kafka consumer processed every message exactly once for months, and then one afternoon a handful of orders shipped twice. Nothing had crashed, no message was redelivered by a broker retry, and the offsets on the dashboard looked healthy. The consumer had committed an offset it had not actually finished processing, a rebalance handed the same partition to another instance, and that instance dutifully reprocessed a batch the first one was still working through.
A database failover completed in under a minute. DNS flipped the endpoint to the new primary's address almost immediately, every other service reconnected, and the incident should have been over. One Java service kept hammering the old, now demoted, host for the better part of an hour. It was not a bug in our code or a slow health check. The JVM had resolved that hostname once at startup and cached the IP address effectively forever.
A lock-free stack built on compare-and-swap passed every load test and then corrupted itself in production, popping a node that had already been freed. The CAS never failed, and a failing CAS was the only case anyone had thought to worry about. The bug was a value swinging from A to B and back to A while a paused thread was not looking, so the compare saw the same bit pattern and happily proceeded on data that was no longer the data it started with.
Each topic page collects every article on that subject.