Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache-kafka one of the nodes is shutting down automatically

Highlighted

Apache-kafka one of the nodes is shutting down automatically

New Contributor


We are having a setup of 3 zookeepers and 3 kafka in our requirement. Recently , we faced issue like one of the kafka services is shutting down automatically . When we checked in depth , the same is getting down frequently every month on the same date and there are no scripts or cronjobs enabled in the server. Also there are no trace of logs why it is getting shutdown . Could you please let us know why it is getting down frequently.

The only logs we can see other than shutting down has been attached here.

[2020-03-05 23:50:42,004] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)

[2020-03-05 23:50:42,014] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)

[2020-03-05 23:50:42,781] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,8] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:42,781] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,2] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:42,781] ERROR [ReplicaFetcherThread-0-1]: Error for partition [alert1,1] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:42,782] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,5] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:43,785] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,5] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:43,785] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,2] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:43,785] ERROR [ReplicaFetcherThread-0-1]: Error for partition [alert1,1] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:43,785] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,8] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:44,789] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,5] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:44,789] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,2] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:44,789] ERROR [ReplicaFetcherThread-0-1]: Error for partition [alert1,1] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:44,789] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,8] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:46,194] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,5] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:46,194] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,2] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:46,194] ERROR [ReplicaFetcherThread-0-1]: Error for partition [alert1,1] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:46,194] ERROR [ReplicaFetcherThread-0-1]: Error for partition [trip1,8] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-03-05 23:50:46,700] INFO Partition [trip1,3] on broker 2: Shrinking ISR from 1,2,0 to 1,2 (kafka.cluster.Partition)

[2020-03-05 23:50:46,703] INFO Partition [trip1,3] on broker 2: Cached zkVersion [868] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)


Kafkaprd3_1.png

Don't have an account?
Coming from Hortonworks? Activate your account here