Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

KAFKA - how to re-balance the partitions to available brokers

avatar
author-rank mike_bronson7
Guru

Created ‎09-29-2018 07:20 PM

hi all

we have ambari cluster , HDP version - 2.6.4 with 3 kafka machines and 3 zookeeper's

kafka1 - 1001

kafka2 - 1002

kafka3 - 1003

we issue is about to re-balance the partitions to available brokers , we can see ( down ) that some partitions under-replicated with only two brokers in isr. Instead with 3

1. what is the best way to re balance the kafka topic partitions in ISR

2. second issue is that leader - 1002 is missing - how to fix it ?

remark - we have 23 topics ( the list down is partial )

example what we get :

[kafka@kafka01 bin]$ ./kafka-topics.sh -describe --zookeeper master:2181
Topic:lop_gt        PartitionCount:50       ReplicationFactor:3     Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
        Topic: lop_gt       Partition: 0    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 1    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 2    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 3    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 4    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 5    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 6    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 7    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 8    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 9    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 10   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 11   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 12   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 13   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 14   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 15   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 16   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 17   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 18   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 19   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 20   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 21   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 22   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 23   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 24   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 25   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 26   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 27   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 28   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 29   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 30   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 31   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 32   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 33   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 34   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 35   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 36   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 37   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 38   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 39   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 40   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 41   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 42   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 43   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 44   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 45   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 46   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 47   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 48   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 49   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
Topic:_schemas  PartitionCount:1        ReplicationFactor:3     Configs:cleanup.policy=compact
        Topic: _schemas Partition: 0    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
Topic:ambari_kafka_service_check        PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: ambari_kafka_service_check       Partition: 0    Leader: 1002    Replicas: 1002  Isr: 1002
Topic:jr_dfse    PartitionCount:10       ReplicationFactor:3     Configs:
        Topic: jr_dfse   Partition: 0    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: jr_dfse   Partition: 1    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 2    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 3    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: jr_dfse   Partition: 4    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 5    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 6    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: jr_dfse   Partition: 7    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 8    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 9    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
Topic:frte_nnc   PartitionCount:6        ReplicationFactor:3     Configs:
        Topic: frte_nnc  Partition: 0    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: frte_nnc  Partition: 1    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: frte_nnc  Partition: 2    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: frte_nnc  Partition: 3    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: frte_nnc  Partition: 4    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: frte_nnc  Partition: 5    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
Topic:erw_plk    PartitionCount:100      ReplicationFactor:3     Configs:
        Topic: erw_plk   Partition: 0    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 1    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 2    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 3    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 4    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 5    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 6    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 7    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 8    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 9    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 10   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 11   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 12   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 13   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 14   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 15   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 16   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 17   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 18   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 19   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 20   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 21   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 22   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 23   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 24   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 25   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 26   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 27   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 28   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 29   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 30   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 31   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 32   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 33   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 34   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 35   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 36   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 37   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 38   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 39   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 40   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 41   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 42   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 43   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 44   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 45   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 46   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 47   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 48   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 49   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 50   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 51   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 52   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 53   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 54   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 55   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 56   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 57   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 58   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 59   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 60   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 61   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 62   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 63   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 64   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 65   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 66   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 67   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 68   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 69   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 70   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 71   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 72   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 73   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 74   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 75   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 76   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 77   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 78   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 79   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 80   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 81   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 82   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 83   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 84   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 85   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 86   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 87   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 88   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 89   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 90   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 91   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 92   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 93   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 94   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 95   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 96   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 97   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 98   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 99   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
Topic:loe_sd        PartitionCount:6        ReplicationFactor:3     Configs:
        Topic: loe_sd       Partition: 0    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: loe_sd       Partition: 1    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: loe_sd       Partition: 2    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: loe_sd       Partition: 3    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: loe_sd       Partition: 4    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: loe_sd       Partition: 5    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
Michael-Bronson
7,176 Views
0 Kudos
2 REPLIES 2

avatar
author-rank ayes
New Contributor

Created ‎12-03-2020 11:34 PM

Hi,

Any solution provided to the second issue ?

 

second issue is that leader - 1002 is missing - how to fix it 

5,933 Views
0 Kudos

avatar
author-rank Shelton
Master Mentor

Created on ‎12-16-2020 02:28 PM - edited ‎12-16-2020 02:37 PM

@mike_bronson7 

To achieve your goal for the 2 issues you will need to edit server.properties of Kafka to add the following line.

 

auto.leader.rebalance.enable = false

 

Then run the below assuming you are having a zookeeper quorum of host1;host2,host3

 

bin/kafka-preferred-replica-election.sh --zookeeper host1:2181,host2:2181,host3:2181/kafka

 

This should balance your partitions you can validate with 

bin/kafka-topics.sh --zookeeper host1:2181,host2:2181,host3:2181/kafka --describe


For the second issue with the lost  broker, you need to create a new broker and update the broker.id with the previous broker's id which was not gone or not recoverable then run

$ kafka-preferred-replica-election.sh

to balance the topics.

5,881 Views
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streams Messaging - Kubernetes Operator ...
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics 1.14 for Cloudera Pu...
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
View More Announcements