Redis Replication Lag Spike: The PSYNC Buffer Overflow That Stalled Primary Writes

In high-throughput Redis deployments, few operational surprises are more disruptive than a routine replica reconnect triggering a cascading primary freeze. The system starts with a brief network jitter or a packet pause between availability zones. Seconds later, primary write latencies jump from 200 microseconds to several hundred milliseconds, memory consumption balloons, and downstream services face connection timeouts.

The root cause is almost always an undersized repl-backlog-size combined with aggressive client traffic. When a replica drops its connection and reconnects, it presents its replication offset via the PSYNC protocol. If that offset has fallen off the edge of the primary circular backlog buffer, partial resynchronization fails, forcing Redis into a heavy Full Resynchronization (Full Sync).

1. Anatomy of PSYNC and the Replication Backlog

Redis maintains an in-memory circular ring buffer known as the replication backlog. As clients write to the primary, Redis appends raw replication stream commands to this buffer and increments a global 64-bit replication offset (master_repl_offset).

[ ... data chunk ... | older offset <--- repl-backlog-size ---> current offset ]
                                ^
                                Replica reconnects here:
                                Offset present in backlog -> Partial Sync (PSYNC) OK
[ older offset ... repl-backlog-size ] <=== Reconnect offset was here (DROPPED!)
                                     -> Full Sync (BGSAVE / SYNC) REQUIRED!

When replica R reconnects after a 15-second network hiccup, it sends:

PSYNC <master_replid> <replica_offset>

If replica_offset >= (master_repl_offset - repl-backlog-size), the primary replies with +CONTINUE and streams only the missing command bytes. The resync completes in milliseconds with zero disk I/O.

However, if the write volume during those 15 seconds exceeded the backlog size, the replica offset is gone. The primary replies with +FULLRESYNC <replid> <offset> and must transmit the entire database image from scratch.

2. The Cascading Failure: Fork, Copy-on-Write, and I/O Thrashing

Once a Full Sync is triggered, the primary must generate a point-in-time snapshot. Historically, this meant invoking fork() to spawn a child process that writes a background RDB file (BGSAVE). Under modern production loads, this introduces three distinct failure vectors:

• Fork Execution Latency: While fork() does not copy physical pages immediately, the kernel must duplicate the page table entries for the parent process. On an instance with 32 GB of RAM and 4 KB page sizes, duplicating page tables can pause the single-threaded Redis event loop for 100 to 400 milliseconds. During this pause, all client commands wait.
• Copy-on-Write (COW) Memory Inflation: As write traffic continues to hit the primary while the background child process writes to disk, Linux kernel pages modified by the parent must be physically copied. In write-heavy workloads, memory usage can surge by 30% to 80%, tripping OS out-of-memory killers (OOM) or triggering system swap thrashing.
• Client Output Buffer Overflow (client-output-buffer-limit slave): While the primary generates and transfers the multi-gigabyte RDB dump, new incoming writes accumulate in the replica dedicated client output buffer. If that buffer exceeds client-output-buffer-limit slave, Redis abruptly disconnects the replica. The replica reconnects, restarts the Full Sync loop, and locks the primary into perpetual BGSAVE storms.

3. Sizing the Backlog Ring Buffer Mathematically

The default repl-backlog-size in standard Redis configurations is often a meager 1 MB. In an environment processing 10,000 writes per second at 500 bytes per command, 1 MB provides less than 0.2 seconds of buffer room before partial resync becomes impossible.

The formula for safe backlog sizing must account for peak write rate and reasonable network partition durations:

repl-backlog-size = peak_write_bytes_per_second * target_disconnect_tolerance_seconds

For example, if peak write throughput is 15 MB/s and you want replicas to tolerate up to 120 seconds of network degradation, maintenance failover, or route re-convergence:

repl-backlog-size = 15 MB/s * 120 s = 1800 MB (~1.8 GB)

Allocating 2 GB to the replication backlog on a 32 GB server is cheap insurance against ever suffering an unwanted Full Sync.

4. Modern Mitigations in Redis

Diskless Replication (repl-diskless-sync)

Instead of writing the RDB file to slow disk storage and then streaming it from disk to the socket, diskless replication forks a process that writes the serialized RDB data directly to the replica network sockets:

repl-diskless-sync yes
repl-diskless-sync-delay 5

The repl-diskless-sync-delay setting gives Redis a few seconds to wait for additional replicas to connect before initiating the stream, allowing one single fork pass to feed multiple replicas simultaneously.

Increasing Slave Buffer Limits

To prevent the primary from severing the replica during large data transfers, tune the slave output buffer limits to handle your maximum expected write volume during the sync window:

# Syntax: client-output-buffer-limit slave <hard-limit> <soft-limit> <soft-seconds>
client-output-buffer-limit slave 2048mb 512mb 120

Replication ID Handover (PSYNC2)

Modern Redis versions support PSYNC2, maintaining two replication IDs (master_replid and master_replid2). When a replica is promoted to primary during a failover, remaining replicas can perform partial resynchronization against the new primary without needing a Full Sync, provided their offsets align with the common ancestor point.

Key Takeaways

• Never leave repl-backlog-size at its default in production. Size it to cover at least 2 to 5 minutes of peak write bandwidth.
• Monitor sync_full and sync_partial_ok metrics in INFO replication. Any spike in sync_full indicates an undersized backlog or buffer limit breach.
• Enable diskless replication to eliminate disk write bottlenecks during inevitable resync operations.