For this partition:
Topic: truckevent Partition: 0 Leader: -1 Replicas: 57 Isr: 57
Since you only have replication-factor of 1, you don't have another replica to bring up as the leader. In this instance, you can move the replica to a running broker, but you will have to enable unclean leader election, as that new replica isn't in the ISR for partition 0. You won't have any of the data that is in that partition, as it all resides on broker 57. If you can move that data to broker 58, you would preserve the messages in that partition.
If you need to go that route, you would need to use the kafka-reassign-partitions command to take the single partition (truckevent partition 0 on broker 57) and move it to another broker. See the full documentation here [1]
In this example, we are moving partition 0 for the testmove topic from broker 52 to 51 in this example:
[root@host-1 ~]# kafka-topics --describe --topic testmove --zookeeper ${HOSTNAME}:2181
Topic:testmove PartitionCount:2 ReplicationFactor:2 Configs:
Topic: testmove Partition: 0 Leader: -1 Replicas: 52 Isr: 52
Topic: testmove Partition: 1 Leader: 50 Replicas: 50 Isr: 50
1. Create the topictomove.json file:
echo '{"topics": [{"topic": "testmove"}], "version":1 }' > topictomove.json
2. generate the partitions to reassign using the --generate option:
root@host-1 ~]# kafka-reassign-partitions --zookeeper ${HOSTNAME}:2181 --generate --broker-list 51,50 --topics-to-move-json-file topictomove.json
Current partition replica assignment
{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[52]},{"topic":"testmove","partition":1,"replicas":[50]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[50]},{"topic":"testmove","partition":1,"replicas":[51]}]}
3. Take the output from the "Proposed partition reassignment configuration" and include just the partition you want to move, in a file called reassign.json. In this instance, we are moving partition0 from replica 52 to replica 51 so the broker id listed in the json file will be the new target:
[root@host-1 ~]# cat reassign.json
{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[51]}]}
4. If possible, copy the partition folder from the invalid broker to the new broker. This will ensure the partition contains all the data on the new broker:
rsync -ave ssh /var/local/kafka/data/testmove-0 broker51:/var/local/kafka/data/
5. Use the --execute command to make the changes:
[root@host-1 ~]# kafka-reassign-partitions --zookeeper ${HOSTNAME}:2181 --execute --reassignment-json-file reassign.json
Current partition replica assignment
{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[52]},{"topic":"testmove","partition":1,"replicas":[50]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[51]}]}
6. At this time, the new replica will show up, but its not in the ISR list, and it can't be elected leader:
[root@host-1 ~]# kafka-topics --describe --topic testmove --zookeeper ${HOSTNAME}:2181
Topic:testmove PartitionCount:2 ReplicationFactor:2 Configs:
Topic: testmove Partition: 0 Leader: -1 Replicas: 51,52 Isr: 52
Topic: testmove Partition: 1 Leader: 50 Replicas: 50 Isr: 50And in the logs on the new broker, you will see the following:
2016-08-05 10:45:17,119 ERROR state.change.logger: Broker 51 received LeaderAndIsrRequest with correlation id 126 from controller 50 epoch 4 for partition [testmove,0] but cannot become follower since the new leader -1 is unavailable.
7. You will have to enable unclean.leader.election.enable in Cloudera Manager for the Kafka service configuration, and restart the kafka service. The broker that is the active controller needs to be restarted with this flag enabled to properly allow the new leader to be elected, even though it is not in the ISR list.
7. The new replica is now shown as the new leader:
[root@host-1 ~]# kafka-topics --describe --topic testmove --zookeeper ${HOSTNAME}:2181
Topic:testmove PartitionCount:2 ReplicationFactor:1 Configs:
Topic: testmove Partition: 0 Leader: 51 Replicas: 51 Isr: 51
Topic: testmove Partition: 1 Leader: 50 Replicas: 50 Isr: 50
8. Turn off unclean.leader.election.enable and restart the cluster to ensure you have the safeguard back in place.
NOTE: If you had data in the old partition on replica that wouldn't start,unless you copy that data from the old broker (as noted in step 4,) you will lose messages that were on that partition.
I would recommend adding more replicas per partition. If you had a replication factor of 2 or more, you could move the replica to another server using the kafka-reassign-partitions command without having to enable unclean leader election (replication would happen automatically from the leader), but in your case, you only have 1 replica per partition, so you must enable the unclean leader election.
[1] https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignParti...
