Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Unable to create topic in kafka cluster.

avatar
Contributor

Dear Team, 

Problem Statement: While creating new topics ISR for all the topics getting shrink automatically, even though we created topic it was throwing below errors in kafka server.log.

VERSIONS:

  1. Kafka version (0.10.0),
  2. HDP-2.6.3.0
  3. No. of Brokers 3 with 16GB JVM Heap.

We have tried below steps as per the URL but no luck 😞

https://medium.com/@nblaye/reset-consumer-offsets-topic-in-kafka-with-zookeeper-5910213284a2

  1. We stop all brokers.
  2. Removed below files from all brokers.

cleaner-offset-checkpoint , .lock , recovery-point-offset-checkpoint , replication-offset-checkpoint

  1. Restarted all brokers.

Logs for reference found in server.log

[2020-12-24 00:22:23,481] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,9] to broker 1002:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-12-24 00:18:59,924] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,36] to broker 1002:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread)
[2020-12-24 00:18:59,951] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,45] to broker 1002:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread)


[2020-12-24 01:14:20,923] INFO Partition [__consumer_offsets,14] on broker 1003: Shrinking ISR for partition [__consumer_offsets,14] from 1002,1003,1001 to 1002,1003 (kafka.cluster.Partition)
[2020-12-24 01:14:20,925] INFO Partition [__consumer_offsets,32] on broker 1003: Shrinking ISR for partition [__consumer_offsets,32] from 1002,1003,1001 to 1002,1003 (kafka.cluster.Partition)
[2020-12-24 01:14:20,927] INFO Partition [__consumer_offsets,29] on broker 1003: Shrinking ISR for partition [__consumer_offsets,29] from 1003,1002,1001 to 1003,1002 (kafka.cluster.Partition)
[2020-12-24 01:14:20,928] INFO Partition [__consumer_offsets,44] on broker 1003: Shrinking ISR for partition [__consumer_offsets,44] from 1003,1002,1001 to 1003,1002 (kafka.cluster.Partition)

 

Please suggest on this and Thanks in advance.

6 REPLIES 6

avatar
Master Mentor

@rohit_r_sharma 
Can you share the syntax for the topic creation? Is you cluster kerberized?  Is your Ranger Kafka plugin enabled? 
Please response and tag me!

avatar
New Contributor

@Shelton ,

-Syntax of topic creation : 
kafka-topics.sh --create --topic "topic_name" --partitions 1 --replication-factor 2 --zookeeper broker_Fqdn1:2181,broker_Fqdn2:2181,broker_Fqdn3:2181

-Yes my cluster is kerberized.

-Yes ranger Kafka plugin is enabled

Thanks in advance.

avatar
Master Mentor

@rohit_sharma 

 

Can you change your syntax as below, note the zookeeper ensemble

/bin/kafka-topics.sh --create \
--zookeeper zk1:2181,zk2:2181,zk3:2181 \
--topic "topic_name" \
--partitions 1> \
--replication-factor 2

Hope that helps

 

avatar
Master Mentor

@rohit_sharma 
Any updates? Do you still need help?

avatar
New Contributor

@Shelton ,
sorry for replying late, 

The syntax you gave is the same we used to create the topic, but one doubt is the ">" which is after 1 is typo error or the valid syntax. Also, we are using zk to create topics.

--partitions 1> \

 Still, i can see the error in Kafka:

 WARN [ReplicaFetcherThread-0-1001], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@11c4fb92 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 1001 was disconnected before the response was read

Thankyou for help in advance

avatar
Master Mentor

@rohit_sharma 

Indeed > was a typo

Can you share your code here ?