The Failover That the JVM Slept Through: A DNS Record Nobody Re-Resolved

Our managed database did a failover during a routine maintenance event. The provider promoted a standby, and within seconds the database endpoint's DNS record pointed at the new primary. Most of the fleet reconnected on the next connection attempt and moved on. One Java service did not. It kept opening connections to the old host, which was now a demoted replica that rejected writes, and it stayed stuck on that address long after DNS had been updated, throwing write errors while every other service was healthy. Nothing in our code pinned an IP. We connected by hostname, exactly like everyone else.

The address the JVM never let go of

When Java resolves a hostname through InetAddress, it caches the result. The lifetime of that cache is controlled by a security property, networkaddress.cache.ttl, and its default behavior is the surprising part. If a security manager is installed, the default is to cache successful lookups forever (a TTL of -1). Without a security manager, the JDK falls back to an implementation default that has historically been a small number of seconds but is not something you want to rely on by accident. In our case an old base image set networkaddress.cache.ttl=-1 explicitly in java.security, a well-intentioned relic from someone who once wanted to avoid repeated DNS lookups. The consequence: the first time the service resolved the database hostname, at startup, it burned that IP into memory and never asked DNS again.

# $JAVA_HOME/conf/security/java.security
# The line that caused it:
networkaddress.cache.ttl=-1     # cache forever; never re-resolve
# What we actually wanted:
networkaddress.cache.ttl=30     # re-resolve at most every 30s

So when the endpoint flipped, DNS was correct within seconds, the OS resolver knew the new address, dig on the host returned the new address, but the JVM inside the container was still handing out the address it had cached at boot. It was not ignoring a short TTL; it had been told never to expire the entry at all.

Why the connection pool made it worse

Even a sane DNS TTL would not have been enough on its own, and this is the part people miss. A connection pool holds already-established TCP connections to a resolved IP. DNS resolution happens when a new connection is created, not on every query over an existing one. Our pool was configured to keep connections alive for a long time with a generous max lifetime, so even after the old host started rejecting writes, the pool kept reusing sockets pointed at it and only slowly created new ones. The two effects compounded: the JVM would not re-resolve the name, and the pool would not cycle the sockets that were pinned to the stale address. Fixing DNS caching without also bounding connection lifetime would have left us waiting on the pool; fixing the pool without DNS caching would have left new connections resolving to the same cached IP.

What it looked like on the dashboard

The confusing signal was that DNS was fine. Every diagnostic we reached for pointed at a healthy DNS system: the record had the right value, the TTL on the record was short, other languages' clients on the same hosts had already failed over. Ping and dig from inside the same container returned the new IP. The only thing still using the old address was the JVM's own internal cache, which does not show up in any external tool. We spent the first twenty minutes convinced the database failover itself had not completed, because from the application's perspective it was still talking to the demoted host.

The fix, in two parts

First, bound the DNS cache. Set networkaddress.cache.ttl to a small positive value so the JVM re-resolves on a sane cadence. This is a security property, so it is set in java.security or via java.security.Security.setProperty early in startup, not as an ordinary -D system property, which is a common mistake, the -D form silently does nothing for this one.

// Set early, before any InetAddress resolution happens:
java.security.Security.setProperty("networkaddress.cache.ttl", "30");
java.security.Security.setProperty("networkaddress.cache.negative.ttl", "5");

Second, bound connection lifetime in the pool so sockets pinned to a departed address get retired promptly. In HikariCP that is maxLifetime, kept comfortably under any upstream idle timeout, so the pool naturally recycles connections and picks up freshly resolved addresses.

hikari.setMaxLifetime(Duration.ofMinutes(10).toMillis());
// plus validation so a broken/stale connection is discarded, not handed out:
hikari.setKeepaliveTime(Duration.ofSeconds(30).toMillis());

Together these give a bounded worst-case failover time: at most the DNS TTL to learn the new address, plus at most the connection max-lifetime for the pool to stop reusing the old sockets. Both are now numbers we chose deliberately instead of "forever" chosen for us by an old config file.

Also worth knowing: the negative cache

There is a sibling property, networkaddress.cache.negative.ttl, that caches failed lookups. Its default is short but non-zero, and if a name is temporarily unresolvable during a deploy, a too-long negative TTL will keep the service failing to resolve a name that has since come back. It bit us as a smaller aftershock during the same incident, so we pinned it explicitly too.

Why it hid

DNS-based failover is the kind of thing that works in every test because tests never keep a JVM alive across an endpoint change, they start fresh, resolve once, and finish. The forever-cache only hurts a long-lived process that resolved a name before the name changed underneath it, which is exactly a production service during a failover and never a CI run. And because the misconfiguration lived in a base image's java.security file, it was invisible in the application repo entirely; nobody who read the service's own code could have found it. The symptom pointed at DNS, the cause was the one component that had stopped listening to DNS.

Rules of thumb

• The JVM caches DNS resolutions, and with a security manager the default is to cache successful lookups forever. Set networkaddress.cache.ttl to a small positive value explicitly; do not inherit "forever" from a base image.
• It is a security property. Set it in java.security or via Security.setProperty early in startup. The -Dnetworkaddress.cache.ttl system-property form does not work.
• DNS TTL alone is not enough. Connection pools reuse sockets pinned to already-resolved IPs, so bound maxLifetime too, or the pool will keep talking to a departed host well after the name has moved.
• Your failover budget is DNS TTL plus connection max-lifetime. Pick both numbers on purpose so the worst case is bounded and known.
• When failover "does not happen" but every external DNS tool looks healthy, suspect a client-side cache, the JVM's or the pool's, not the DNS system itself.