Dear Team,
Problem Statement: While creating new topics ISR for all the topics getting shrink automatically, even though we created topic it was throwing below errors in kafka server.log.
VERSIONS:
- Kafka version (0.10.0),
- HDP-2.6.3.0
- No. of Brokers 3 with 16GB JVM Heap.
We have tried below steps as per the URL but no luck
https://medium.com/@nblaye/reset-consumer-offsets-topic-in-kafka-with-zookeeper-5910213284a2
- We stop all brokers.
- Removed below files from all brokers.
cleaner-offset-checkpoint , .lock , recovery-point-offset-checkpoint , replication-offset-checkpoint
- Restarted all brokers.
Logs for reference found in server.log
[2020-12-24 00:22:23,481] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,9] to broker 1002:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2020-12-24 00:18:59,924] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,36] to broker 1002:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread)
[2020-12-24 00:18:59,951] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,45] to broker 1002:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread)
[2020-12-24 01:14:20,923] INFO Partition [__consumer_offsets,14] on broker 1003: Shrinking ISR for partition [__consumer_offsets,14] from 1002,1003,1001 to 1002,1003 (kafka.cluster.Partition)
[2020-12-24 01:14:20,925] INFO Partition [__consumer_offsets,32] on broker 1003: Shrinking ISR for partition [__consumer_offsets,32] from 1002,1003,1001 to 1002,1003 (kafka.cluster.Partition)
[2020-12-24 01:14:20,927] INFO Partition [__consumer_offsets,29] on broker 1003: Shrinking ISR for partition [__consumer_offsets,29] from 1003,1002,1001 to 1003,1002 (kafka.cluster.Partition)
[2020-12-24 01:14:20,928] INFO Partition [__consumer_offsets,44] on broker 1003: Shrinking ISR for partition [__consumer_offsets,44] from 1003,1002,1001 to 1003,1002 (kafka.cluster.Partition)
Please suggest on this and Thanks in advance.
