KAFKA-19193 Support rack-aware partitioning for Kafka producer by ivanyu · Pull Request #19850 · apache/kafka

ivanyu · 2025-05-29T17:02:10Z

According to
KIP-1123,
this commit adds the support for rack-aware partitioning to
BuiltInPartitioner. It comes with two new configs for the producer:
partitioner.rack.aware and client.rack, which allows enabling the
new behavior.

Apart from the added unit tests, the desired behavior was tested by
kafka-producer-perf-test.sh with an existing and a non-existing rack
against a 4 node cluster with two racks and 12-partition topic:

./kafka_2.13-4.1.0-SNAPSHOT/bin/kafka-producer-perf-test.sh \
  --topic test-topic --num-records 100000 --throughput -1 --record-size
1 \
  --producer-props bootstrap.servers=127.0.0.1:9092 \
  client.rack=rack0 partitioner.rack.aware=true

Reviewers: Viktor Somogyi-Vass viktorsomogyi@gmail.com, Liam
Clarke-Hutchinson liam@steelsky.co.nz

github-actions · 2025-06-06T03:28:00Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

marcospassos · 2025-08-21T18:56:01Z

Is there any chance this feature will be included in the next release?

ivanyu · 2025-08-22T14:24:50Z

@marcospassos I'm waiting for 4.1 release to be cut, after that I'll ping the mailing list for review.

github-actions · 2025-08-23T03:21:42Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2025-09-18T03:17:50Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

LiamClarkeNZ

Love the tests.

LiamClarkeNZ · 2025-10-04T04:30:32Z

+    private final String rack;

-    private volatile PartitionLoadStats partitionLoadStats = null;
+    private volatile PartitionLoadStatsHolder partitionLoadStats = null;


Suggested change

private volatile PartitionLoadStatsHolder partitionLoadStats = null;

private volatile PartitionLoadStatsHolder partitionLoadStatsHolder = null;

As it got a bit confusing when later the code is retrieving partitionLoadStats.inThisRack which is a member of PartitionLoadStatsHolder of type PartitionLoadStats. :D

Thanks, renamed

LiamClarkeNZ · 2025-10-04T04:37:03Z


        // Create partitions with "sticky" batch size to accommodate 3 records.
-        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3);
+        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, false, "");


Suggested change

BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, false, "");

final boolean rackAware = false;

final String clientRackId = "";

BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, rackAware, clientRackId);

LiamClarkeNZ · 2025-10-04T04:37:52Z


        // Check that switching works even when there is one partition.
-        BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1);
+        BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, false, "");


Suggested change

BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, false, "");

BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, rackAware, clientRackId);

Just to make the meaning of false and "" obvious at a glance :)

LiamClarkeNZ · 2025-10-04T07:57:18Z

+                partitionRacks[i] = leader.rack();
+            }
+            allPartitions.add(new PartitionInfo(TOPIC_A, i, leader, NODES, NODES));
            expectedFrequencies[i] = 6 - queueSizes[i];  // 6 is max(queueSizes) + 1


Love the comment :)

github-actions · 2026-02-02T04:17:43Z

This PR has been closed since it has not had any activity in 120 days. If you feel like this
was a mistake, or you would like to continue working on it, please feel free to re-open the
PR and ask for a review.

SoerenHenning · 2026-02-06T10:30:21Z

Hi, unfortunately this was closed. Are there any plan to continue this effort or anything we could do to support it?

marcospassos · 2026-02-06T12:33:08Z

This issue would definitely benefit the community and should not be closed.

kirktrue · 2026-03-19T19:08:19Z

@ivanyu—are you still able to work on this? Or do you need the community to follow up? LMK. Thanks!

ivanyu · 2026-03-19T19:51:33Z

@kirktrue sure, I will review the PR soon!

ivanyu · 2026-03-28T16:25:33Z

For some reason, I cannot reopen the PR (the button isn't even there). Maybe someone else can?

ivanyu · 2026-03-30T08:45:52Z

Thank you @LiamClarkeNZ, I addressed your comments

viktorsomogyi · 2026-04-20T15:39:38Z

+            this.rack = rack;
+
+            if (rackAware && Utils.isBlank(rack)) {
+                throw new IllegalArgumentException("client.rack must be provided if partitioner.rack.aware is enabled");


The KIP states:

Select the next partition from all partitions following the current algorithm in the following cases:

if the client rack is not specified;

if there are no partitions with leaders in the same rack as the client;

if all the “nearby” partitions are unavailable.

if partitioner.rack.aware is false.

Based on this, is it correct to throw an exception here?

I think this is OK, for the sake of the user. This exception will be thrown at the producer constructor time, preventing inconsistent config from silently sneaking in.

Should we use ConfigException instead for consistency?

According to KIP-1123, this commit adds the support for rack-aware partitioning to `BuiltInPartitioner`. It comes with two new configs for the producer: `partitioner.rack.aware` and `client.rack`, which allows enabling the new behavior. Apart from the added unit tests, the desired behavior was validated by `kafka-producer-perf-test.sh` with an existing and a non-existing rack against a 4 node cluster with two racks and 12-partition topic: ```shell ./kafka_2.13-4.1.0-SNAPSHOT/bin/kafka-producer-perf-test.sh \ --topic test-topic --num-records 100000 --throughput -1 --record-size 1 \ --producer-props bootstrap.servers=127.0.0.1:9092 \ client.rack=rack0 partitioner.rack.aware=true ```

viktorsomogyi

LGTM, let's merge it when the CI is green.

ivanyu · 2026-04-21T14:25:31Z

The failed tests are marked as flaky and pass locally

chia7712

@ivanyu thanks for driving this patch! I left a few minor suggestions for you. On a related note, have you considered adding a metric like rack-aware-fallback-rate? If it's out of scope for your KIP, I'd gladly draft a follow-up KIP to add it. A silent fallback might lead to an unpleasant surprise for users when they see their cloud networking bills!

chia7712 · 2026-04-22T13:51:40Z

+            this.rack = rack;
+
+            if (rackAware && Utils.isBlank(rack)) {
+                throw new IllegalArgumentException("client.rack must be provided if partitioner.rack.aware is enabled");


Should we use ConfigException instead for consistency?

chia7712 · 2026-04-22T14:07:17Z

        // Invert and fold the queue size, so that they become separator values in the CFT.
+        invertAndFoldQueueSizeArray(queueSizes, maxSizePlus1, length);
+
+        log.trace("Partition load stats for topic {}: CFT={}, IDs={}, length={}",


Should we include the rack-specific load stats (partitionLoadStatsInThisRack) in this trace log when rackAware is true?

github-actions Bot added triage PRs from the community producer clients labels May 29, 2025

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch 11 times, most recently from 1af9189 to 564e60f Compare May 31, 2025 06:14

ivanyu marked this pull request as ready for review May 31, 2025 06:20

github-actions Bot added the needs-attention label Jun 6, 2025

zornhsu mentioned this pull request Jun 16, 2025

feat(producer): rack-aware partitioner IBM/sarama#3198

Closed

github-actions Bot removed the needs-attention label Aug 22, 2025

github-actions Bot added the needs-attention label Aug 23, 2025

showuon self-assigned this Sep 17, 2025

github-actions Bot removed the needs-attention label Sep 17, 2025

github-actions Bot added the needs-attention label Sep 18, 2025

LiamClarkeNZ approved these changes Oct 4, 2025

View reviewed changes

github-actions Bot removed needs-attention triage PRs from the community labels Oct 5, 2025

github-actions Bot added the stale Stale PRs label Jan 3, 2026

github-actions Bot added the closed-stale PRs that were closed due to inactivity label Feb 2, 2026

github-actions Bot closed this Feb 2, 2026

viktorsomogyi reopened this Mar 30, 2026

github-actions Bot added the triage PRs from the community label Mar 30, 2026

viktorsomogyi removed stale Stale PRs closed-stale PRs that were closed due to inactivity labels Mar 30, 2026

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch from 564e60f to 6816a21 Compare March 30, 2026 08:45

viktorsomogyi added ci-approved and removed triage PRs from the community labels Mar 30, 2026

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch 3 times, most recently from 1c6c786 to 520eeed Compare March 30, 2026 13:49

viktorsomogyi reviewed Mar 31, 2026

View reviewed changes

Comment thread clients/src/main/java/org/apache/kafka/clients/producer/internals/BuiltInPartitioner.java

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch from 520eeed to 1a47d94 Compare April 2, 2026 06:03

viktorsomogyi reviewed Apr 20, 2026

View reviewed changes

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch from 1a47d94 to 7404aa9 Compare April 21, 2026 07:25

ivanyu force-pushed the kip-1123-rack-aware-partitioning branch from 7404aa9 to e8965db Compare April 21, 2026 09:15

viktorsomogyi approved these changes Apr 21, 2026

View reviewed changes

viktorsomogyi merged commit a3f1732 into apache:trunk Apr 22, 2026
34 of 36 checks passed

chia7712 reviewed Apr 22, 2026

View reviewed changes

	private volatile PartitionLoadStatsHolder partitionLoadStats = null;
	private volatile PartitionLoadStatsHolder partitionLoadStatsHolder = null;

-        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, false, "");
+        final boolean rackAware = false;
+        final String clientRackId = "";
+        BuiltInPartitioner builtInPartitionerA = new SequentialPartitioner(logContext, TOPIC_A, 3, rackAware, clientRackId);

	BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, false, "");
	BuiltInPartitioner builtInPartitionerB = new SequentialPartitioner(logContext, TOPIC_B, 1, rackAware, clientRackId);

Conversation

ivanyu commented May 29, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 6, 2025

Uh oh!

marcospassos commented Aug 21, 2025

Uh oh!

ivanyu commented Aug 22, 2025

Uh oh!

github-actions Bot commented Aug 23, 2025

Uh oh!

github-actions Bot commented Sep 18, 2025

Uh oh!

LiamClarkeNZ left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 2, 2026

Uh oh!

SoerenHenning commented Feb 6, 2026

Uh oh!

marcospassos commented Feb 6, 2026

Uh oh!

kirktrue commented Mar 19, 2026

Uh oh!

ivanyu commented Mar 19, 2026

Uh oh!

ivanyu commented Mar 28, 2026

Uh oh!

ivanyu commented Mar 30, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viktorsomogyi left a comment

Choose a reason for hiding this comment

Uh oh!

ivanyu commented Apr 21, 2026

Uh oh!

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

ivanyu commented May 29, 2025 •

edited by github-actions Bot

Loading