What does "Rebalancing" mean in Apache Kafka context?

When a new consumer joins a consumer group the set of consumers attempt to "rebalance" the load to assign partitions to each consumer. If the set of consumers changes while this assignment is taking place the rebalance will fail and retry. This setting controls the maximum number of attempts before giving up.

the command for this is: rebalance.max.retries and is set to 4 by default.

also, it might be happening if the following is true:

ZooKeeper session timeout. If the consumer fails to send a heartbeat to ZooKeeper for this period of time it is considered dead and a rebalance will occur.

Hope this helps!

Rebalance is the re-assignment of partition ownership among consumers within a given consumer group. Remember that every consumer in a consumer group is assigned one or more topic partitions exclusively.

A Rebalance happens when:

• a consumer JOINS the group
• a consumer SHUTS DOWN cleanly
• a consumer is considered DEAD by the group coordinator. This may happen after a crash or when the consumer is busy with a long-running processing, which means that no heartbeats has been sent in the meanwhile by the consumer to the group coordinator within the configured session interval
• new partitions are added

Being a group coordinator (one of the brokers in the cluster) and a group leader (the first consumer that joins a group) designated for a consumer group, Rebalance can be more or less described as follows:

• the leader receives a list of all consumers in the group from the group coordinator (this will include all consumers that sent a heartbeat recently and which are therefore considered alive) and is responsible for assigning a subset of partitions to each consumer.
• After deciding on the partition assignment (Kafka has a couple built-in partition assignment policies), the group leader sends the list of assignments to the group coordinator, which sends this information to all the consumers.

This applies to Kafka 0.9, but I'm quite sure for newer versions is still valid.

Consumer rebalance decide which consumer is responsible for which subset of all available partitions for some topic(s). For example, you might have a topic with 20 partitions and 10 consumers; at the end of a rebalance, you might expect each consumer to be reading from 2 partitions. If you shut down 10 of those consumers, you might expect each consumer to have 1 partition after a rebalance has completed. Consumer rebalance is a dynamic partition assignment that can handle automatically by Kafka.

A Group Coordinator is one of the brokers responsible to communicate with consumers to achieve rebalances between consumers.In earlier version Zookeeper stored metadata details but the latest version, it stores on brokers. The consumer coordinators receive heartbeat and polling from all consumers of the consumer groups so be aware of each consumer's heartbeat and manager their offset on partitions.

Group Leader: One of a consumer Group work as group leader which is chosen by the Group coordinator and will responsible for making partition assignment decision on behalf of all consumers in a group.

Rebalance Scenario:

1. Consumer Group subscribes to any topics

2. A Consumer instance could not able to send a heartbeat with a session.heart.beat time interval.

3. Consumer long process exceeds the poll timeout

4. Consumer of Consumer group through exception

5. New partition added.

6. Scaling Up and Down consumer. Added new consumer or remove existing consumer manually for

Consumer Rebalance

Consumer rebalance initiated when consumer requests to join a group or leave a group. The Group Leader receives a list of all active consumers from the Group Coordinator. Group Leader decides partition(s) assigned to each consumer by using PartitionAssigner. Once Group Leader finalize partition assignment it sends assignments list to Group Coordinator which send back this information to all consumer. Group only sends applicable partitions to their consumer not other consumer assigned partitions. Only the Group Leader aware of all consumers and their assigned partitions. After the rebalance is complete, consumers start sending Heartbeat to the Group Coordinator that it's alive. Consumers send an OffsetFetch request to the Group Coordinator to get the last committed offsets for their assigned partitions. Consumers start consuming messaged for newly assigned partition.

State Management

While rebalancing, the Group coordinator set its state to Rebalance and wait for all consumers to re-join the group.

When the Group starts rebalancing, the group coordinator first switches its state to rebalance so that all interacting consumers are notified to rejoin the group. Once rebalance completed Group coordinator create new generation ID and notified to all consumers and group proceed to sync stage where consumers send sync request and go to wait until group Leader finish generating new assign partition. Once consumers received a new assigned partition they moved to a stable stage.

Static Membership

This rebalancing is quite a heavy operation as it required to stop all consumers and wait to get the new assigned partition. On each rebalance always create new generation id means refresh everything. To solve this overhead Kafka 2.3+ introduced Static Membership to reduce unnecessary Rebalance. KIP-345

In Static Membership, the consumer state will persist and on Rebalance the same assignment will get apply. It uses a new group.instance.id to persist member identity. So even in the worst-case scenario member id get reshuffle to assign a new partition but still, the same consumer instance-id will get the same partition assignment

instanceId: A, memberId: 1, assignment: {0, 1, 2}
instanceId: B, memberId: 2, assignment: {3, 4, 5}
instanceId: C, memberId: 3, assignment: {6, 7, 8}

And after the restart:

instanceId: A, memberId: 4, assignment: {0, 1, 2}
instanceId: B, memberId: 2, assignment: {3, 4, 5}
instanceId: C, memberId: 3, assignment: {6, 7, 8}   

Ref:

1. https://www.confluent.io/blog/kafka-rebalance-protocol-static-membership

2. https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances