The Order That Shipped Twice: A Rebalance in the Middle of a Long Batch

We ran a consumer that read order events from Kafka and, for each one, called a shipping API to create a label. It had run cleanly for months: one message in, one label out, offsets advancing smoothly on the dashboard. Then during an afternoon traffic spike a customer reported two labels for one order, then another, then a third. Nothing had crashed. No broker had redelivered anything on a network retry. The consumer group looked healthy, lag was low, and the committed offsets were exactly where you would expect them. And yet a batch of orders had been processed twice.

The setup that looked completely normal

The consumer used the default configuration almost everyone starts with: enable.auto.commit=true, which commits the latest polled offsets on a timer in the background, roughly every auto.commit.interval.ms (5 seconds by default). Each poll() returned a batch of up to max.poll.records messages, and the code looped over the batch calling the shipping API one order at a time.

while (running) {
  var records = consumer.poll(Duration.ofMillis(500));
  for (var record : records) {
    shippingApi.createLabel(record.value()); // slow: a network call each
  }
  // no manual commit: auto-commit handles it in the background
}

The trap is in the interaction between two facts that are individually harmless. First, auto-commit does not commit after each message; it commits, on its timer, the offset of the last message returned by the most recent poll(), whether or not your loop has finished handling that batch. Second, the shipping API had gotten slower under load, so a batch that used to take half a second was now taking closer to a minute to grind through.

What actually happened during the spike

Kafka considers a consumer alive only as long as it calls poll() often enough. If more than max.poll.interval.ms (5 minutes by default) elapses between polls, the broker decides that consumer is stuck, kicks it out of the group, and triggers a rebalance that reassigns its partitions to the remaining members. Under the spike, one instance picked up a large batch, and the shipping API was slow enough that the loop ran long. Meanwhile the background auto-committer had already fired on its 5-second timer and committed the offset for the whole batch, because that offset was set the moment poll() returned, long before the loop finished.

Here is the exact sequence that shipped the duplicates:

• Instance A polls 400 records. Auto-commit is now primed to commit the offset just past record 400.
• The 5-second auto-commit timer fires while A is only 50 records into the batch. It commits the offset past 400 anyway. The broker now believes records up to 400 are done.
• A is still calling the shipping API in its loop and takes long enough between polls that it blows past a threshold. A rebalance kicks in and the partition moves to instance B.
• B starts consuming from the committed offset, which is past 400. That looks correct, except A never actually finished records 51 through 400. Those labels were never created by A, or worse, A finished some of them after the commit and B redid them.

Depending on the exact timing, you get either gaps (records committed but never processed) or duplicates (records processed by A after the commit and again by B). We happened to notice the duplicates because they cost money and generated angry customers. The gaps, the orders that silently never shipped, we found only afterward by reconciling against the shipping provider.

Why "exactly once" was never true here

The consumer felt exactly-once because in steady state the batch finished well before the commit timer mattered, and a partition never moved. Exactly-once was an accident of timing, not a property of the design. The moment processing time crept up relative to the commit interval and the poll interval, the two independent clocks, the auto-commit timer and the max-poll-interval deadline, drifted into the danger zone, and the guarantee we thought we had evaporated. Auto-commit gives at-most-once or at-least-once depending on where the crash lands, never exactly-once, and the boundary between "committed" and "processed" is invisible in the happy path.

The tempting non-fixes

Bumping max.poll.interval.ms to something huge stops the rebalances, but it also means a genuinely hung consumer sits undetected for that whole window, and it does nothing about auto-commit racing ahead of the loop. Shrinking max.poll.records makes each batch smaller so the loop finishes faster, which narrows the window but does not close it, the same race still exists, just less often, which is the worst kind of fix because it turns a reproducible bug into a rare flaky one.

The real fix: commit what you have actually processed

The correct move is to stop letting a background timer decide what has been processed and instead commit offsets yourself, after the work is durably done. Turn off auto-commit and commit synchronously once the batch (or each record) is truly complete.

// enable.auto.commit = false
while (running) {
  var records = consumer.poll(Duration.ofMillis(500));
  for (var record : records) {
    shippingApi.createLabel(record.value());
  }
  consumer.commitSync(); // only after every label in the batch is created
}

That closes the timer race: nothing is committed until the loop is genuinely finished. But committing after processing means the guarantee is now at-least-once, if the consumer dies after creating labels but before commitSync() returns, the batch replays and you create some labels twice. At-least-once is the honest, achievable default in Kafka, and the way you make it safe is to require the processing step to be idempotent: pass the order id as an idempotency key to the shipping API so a replayed create is a no-op that returns the existing label rather than making a new one.

shippingApi.createLabel(record.value(),
  /* idempotencyKey */ order.id());  // replay -> same label, not a new one

We also stopped doing slow network calls inside the poll loop for large batches. When per-record work is genuinely long, the poll loop should hand records to a bounded worker pool and pause partitions so the loop keeps calling poll() to stay in the group, or the work should be small enough to finish comfortably inside max.poll.interval.ms. The consumer's job is to stay a healthy group member; long blocking work fights that directly.

Why it hid

Every default here is reasonable in isolation. Auto-commit is the documented, convenient default. Five minutes of poll slack is generous. Batching is how you get throughput. The bug lives only in the relationship between processing time, the commit interval, and the poll deadline, and that relationship was fine until a downstream dependency got slower. There was no error, no exception, no redelivery flag to grep for, the offsets on the dashboard were exactly where a correctly-working consumer would put them. It looked right precisely because the broker was doing exactly what it was told, committing offsets it had every reason to believe were done.

Rules of thumb

• Auto-commit commits the offsets from the last poll() on a timer, not when your processing finishes. It never gives you exactly-once; it gives you a race whose outcome depends on where a crash or rebalance lands.
• Design for at-least-once and make the processing step idempotent (an idempotency key, an upsert, a dedup table). This is the achievable guarantee in Kafka, not exactly-once by wishful configuration.
• Turn off auto-commit and commitSync() after the work is durably done when correctness matters. Commit what you have actually processed, never what you have merely polled.
• Keep per-record work short relative to max.poll.interval.ms, or offload it and keep calling poll(). A consumer that blocks too long between polls gets evicted, and its partitions move mid-batch.
• A healthy offset dashboard proves the broker committed, not that you processed. Reconcile against the downstream's own record when a double-execution is expensive.