Kafka Unclean Leader Election: The Silent Data Loss Trap Behind High Availability

In distributed data platforms, system design often forces an explicit choice between availability and consistency. In Apache Kafka, that balance is governed by the partition leader election protocol and the setting unclean.leader.election.enable. While keeping partition leaders online sounds universally desirable to avoid downtime, allowing unclean leader elections can silently destroy committed messages and violate ordering guarantees across downstream databases.

1. The In-Sync Replicas (ISR) Contract

Kafka partitions replicate records across multiple brokers. At any given moment, the partition replicas are categorized into two groups:

• In-Sync Replicas (ISR): Brokers that are actively fetching messages from the leader and whose replication lag is strictly within replica.lag.time.max.ms.
• Out-of-Sync Replicas (OSR): Brokers that have stalled, suffered GC pauses, or fallen behind the leader log end offset (LEO).

When a producer writes with acks=all, the leader only acknowledges the write after all current ISR members have written the record to their local write-ahead log. The point up to which all ISR members agree is designated as the High Watermark (HW). Consumers are only permitted to read records up to the High Watermark.

2. What Happens During an Unclean Leader Election?

Consider a partition with three replicas: Broker A (Leader, LEO=100), Broker B (Follower, LEO=100), and Broker C (Follower, slow network, LEO=60). The ISR list is [A, B].

Broker A (Leader):  [ ... offsets 0 to 100 ... ]  (ISR)
Broker B (Follower):[ ... offsets 0 to 100 ... ]  (ISR)
Broker C (Follower):[ ... offsets 0 to 60  ... ]  (Out of sync)

--- Scenario: Power loss knocks out Rack 1 (Brokers A and B) ---

With Brokers A and B offline, the partition has zero available ISR members. What should Kafka do?

Behavior A: Clean Election Only (unclean.leader.election.enable = false)

The cluster refuses to elect Broker C. The partition becomes unavailable for both reads and writes. Producers receive NOT_ENOUGH_REPLICAS or LEADER_NOT_AVAILABLE errors. System administrators are alerted, but no data is lost. As soon as Broker A or B boots back up, the partition resumes cleanly at offset 100.

Behavior B: Unclean Election Enabled (unclean.leader.election.enable = true)

The controller elects Broker C as the new partition leader because availability is prioritized over correctness. Now disaster strikes in two steps:

1. Immediate Message Drop: Broker C only possesses records up to offset 60. Records 61 through 100, which were acknowledged to producers and potentially consumed by downstream payment processors, are vanished from the active leader view. New incoming messages are written starting at offset 61 on Broker C.
2. Log Truncation upon Reconnect: When Broker A recovers and rejoins the cluster as a follower, it queries Broker C for its Leader Epoch. Recognizing that Broker C is the authoritative leader for epoch N+1, Broker A is forced to truncate its local log down to offset 60 to match Broker C, permanently erasing offsets 61-100!

3. Downstream Consequences: Offset Inversion and Phantom Duplicates

The damage caused by log truncation extends far beyond the Kafka broker storage. Modern event-driven pipelines rely on monotonic offsets for deduplication, state tracking, and exactly-once processing:

• Consumer State Desynchronization: If a consumer committed offset 95 before the outage, and the partition reopens under Broker C at offset 61, the consumer will see no new messages until Broker C writes 35 new records. It completely misses new events produced at offsets 61-94.
• Database Drift: If an external datastore was already updated using records from the original offsets 61-100, and new unrelated records are subsequently assigned those same offset numbers under the new leader, deduplication tables keying on (topic, partition, offset) will reject legitimate transactions as duplicates.

4. Production Configurations for Maximum Data Integrity

For transactional systems, payment ledgers, and audit logs, clean elections and strict ISR fencing are non-negotiable. The recommended cluster and topic configuration profile:

# 1. Disable unclean leader election cluster-wide and per topic
unclean.leader.election.enable=false

# 2. Require all in-sync replicas to acknowledge
# On producer side:
acks=all

# 3. Minimum replicas that must write before an ack is returned
# For replication factor 3, set min.insync.replicas to 2:
min.insync.replicas=2

With min.insync.replicas=2 and acks=all, if two out of three brokers fail, the sole surviving broker will reject new producer writes with NotEnoughReplicasException instead of silently accepting writes that cannot be replicated. This preserves total consistency across failure domains.

Summary

Unclean leader election is a legacy trade-off suitable only for telemetry, ephemeral metrics, or non-critical tracking where losing minutes of events is preferable to a temporary pipeline block. For core enterprise backends, keeping unclean.leader.election.enable=false combined with min.insync.replicas=2 guarantees that committed data remains immutable and dependable.