Partitioning Strategy: Keys, Hot Partitions, and Ordering

Kafka partitions are the unit of parallelism, of ordering, and — when you pick wrong — of all your performance problems.

The one rule

Ordering is guaranteed per partition. If two records need to be processed in order, they must land on the same partition. The only way to guarantee that is to give them the same key.

// Orders for the same customer must stay in order:
producer.send(new ProducerRecord<>("orders", customerId, payload));

Hot partitions

The default partitioner hashes the key with murmur2. If 5% of your keys produce 80% of your traffic, five partitions will be saturated while the rest sit idle. Signs you have a hot partition:

• One broker pegged at disk/network limits, others quiet.
• Consumer lag concentrated on one partition.
• kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec shows high skew.

Fixing it

Three approaches, from least to most invasive:

1. Composite keys. Append a bounded suffix: customerId + (ts % 8). You fan out to 8 partitions per customer while keeping related records close.
2. Sticky partitioner with null key when you don't need ordering. Default since 2.4 — it batches records to a single partition per window, which improves compression without skew.
3. Custom partitioner that maps known hot keys to a reserved sub-pool of partitions.

Number of partitions

A partition is also a cost — more partitions mean more file handles, more leader elections, and slower rebalances. A rough planning rule:

partitions = max(
  target_throughput / per_partition_throughput,
  max_consumers_in_group
)

And never reduce partition count in place — you can only add. Plan for 2–3× growth from day one.