ML
Engineering Journal

Writing as a second form of thinking

Clear, practical essays on how real software gets designed, breaks, and ships. The failure modes I've run into, the trade-offs that turned out to matter, and the patterns that hold up in production.

Apache Kafka
Jul 06, 20269 min

The Order That Shipped Twice: A Rebalance in the Middle of a Long Batch

A Kafka consumer processed every message exactly once for months, and then one afternoon a handful of orders shipped twice. Nothing had crashed, no message was redelivered by a broker retry, and the offsets on the dashboard looked healthy. The consumer had committed an offset it had not actually finished processing, a rebalance handed the same partition to another instance, and that instance dutifully reprocessed a batch the first one was still working through.

KafkaConsumers
Read
Java / Spring
Jul 06, 20268 min

The Failover That the JVM Slept Through: A DNS Record Nobody Re-Resolved

A database failover completed in under a minute. DNS flipped the endpoint to the new primary's address almost immediately, every other service reconnected, and the incident should have been over. One Java service kept hammering the old, now demoted, host for the better part of an hour. It was not a bug in our code or a slow health check. The JVM had resolved that hostname once at startup and cached the IP address effectively forever.

JavaDNS
Read
Concurrency
Jul 02, 20269 min

The ABA Problem: When Compare-And-Swap Said Nothing Changed and It Lied

A lock-free stack built on compare-and-swap passed every load test and then corrupted itself in production, popping a node that had already been freed. The CAS never failed, and a failing CAS was the only case anyone had thought to worry about. The bug was a value swinging from A to B and back to A while a paused thread was not looking, so the compare saw the same bit pattern and happily proceeded on data that was no longer the data it started with.

ConcurrencyLock-Free
Read
System Design
Jul 02, 20268 min

Token Bucket vs Leaky Bucket: The Rate Limiter That Let the Burst Through

A rate limiter rejected traffic smoothly in every test and then let a client through at ten times its contracted rate during a real burst, taking down a downstream dependency that had no defenses of its own. The limiter was not broken. It was a token bucket, correctly implemented, doing exactly what a token bucket is designed to do, which turned out to be the wrong algorithm for a downstream that could not absorb bursts at all.

System DesignRate Limiting
Read
PostgreSQL
Jun 29, 20268 min

The Query That Quietly Stopped Using Its Index: An Implicit Cast Story

One endpoint went from 4 ms to 1.2 seconds with no code change and no schema change. The index was still there, the query still looked identical, and EXPLAIN insisted on a sequential scan over 40 million rows. The cause was a single implicit type cast that Postgres could not push through the index, introduced by a client library that started sending a parameter as text.

PostgresPerformance
Read
Kubernetes
Jun 29, 20268 min

OOMKilled at 60% Heap: When the JVM Can't See the Container's Limit

A Java service kept getting OOMKilled by Kubernetes while its own heap graphs sat calmly at 60% with gigabytes to spare. The JVM was healthy by every metric it reported. The kernel killed it anyway, because the memory the JVM does not count, off-heap buffers, thread stacks, and metaspace, pushed the whole process past a container limit the JVM was never looking at.

KubernetesJVM
Read
PostgreSQL
Jun 28, 20268 min

The Database Went Read-Only at 4 AM: Transaction ID Wraparound

Postgres started rejecting every write with 'database is not accepting commands to avoid wraparound data loss'. No disk was full, no lock was held, replication was fine. We had quietly burned through the 32-bit transaction ID counter on one table because a long-running process had been blocking vacuum from doing the one job that keeps that counter from running out.

DatabasesPostgres
Read
Concurrency
Jun 28, 20269 min

The Lock That Held Two Owners: A GC Pause Versus a Redis TTL

We used a Redis lock with a TTL to make sure exactly one worker processed each payout. For months it worked, then a batch ran twice and double-charged a handful of accounts. The lock was doing its job perfectly. The problem was that a stop-the-world GC pause outlived the lock's expiry, so a worker that believed it still held the lock kept working while a second worker had already taken it.

ConcurrencyDistributed Systems
Read
TypeScript
Jun 27, 20268 min

The IDs That Came Back Wrong: JavaScript's 2^53 Integer Ceiling

A handful of users reported that clicking a notification took them to the wrong record. The IDs were off by one or two at the very end, always on the biggest accounts. The backend was sending the correct 64-bit ID every time. Our frontend was quietly rounding it the instant it parsed the JSON, because in JavaScript every number is a float.

TypeScriptJavaScript
Read

Browse by topic

Each topic page collects every article on that subject.

Apache Kafka7Redis7Elasticsearch6Kibana5PostgreSQL10Database6Docker4Kubernetes6Observability6System Design10Microservices3Concurrency6Scaling5Security5Testing3AWS · Cloud3React6Node.js6Java / Spring7Next.js3TypeScript5AI · LLM4