Unable to create topic in kafka cluster.

[#1]

Dear Team, 

Problem Statement: While creating new topics ISR for all the topics getting shrink automatically, even though we created topic it was throwing below errors in kafka server.log.

VERSIONS:

1. Kafka version (0.10.0),
2. HDP-2.6.3.0
3. No. of Brokers 3 with 16GB JVM Heap.

We have tried below steps as per the URL but no luck 😞

https://medium.com/@nblaye/reset-consumer-offsets-topic-in-kafka-with-zookeeper-5910213284a2

1. We stop all brokers.
2. Removed below files from all brokers.

cleaner-offset-checkpoint , .lock , recovery-point-offset-checkpoint , replication-offset-checkpoint

1. Restarted all brokers.

Logs for reference found in server.log

[2020-12-24 00:22:23,481] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,9] to broker 1002:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

[2020-12-24 00:18:59,924] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,36] to broker 1002:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread)
[2020-12-24 00:18:59,951] ERROR [ReplicaFetcherThread-0-1002], Error for partition [__consumer_offsets,45] to broker 1002:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread)

[2020-12-24 01:14:20,923] INFO Partition [__consumer_offsets,14] on broker 1003: Shrinking ISR for partition [__consumer_offsets,14] from 1002,1003,1001 to 1002,1003 (kafka.cluster.Partition)
[2020-12-24 01:14:20,925] INFO Partition [__consumer_offsets,32] on broker 1003: Shrinking ISR for partition [__consumer_offsets,32] from 1002,1003,1001 to 1002,1003 (kafka.cluster.Partition)
[2020-12-24 01:14:20,927] INFO Partition [__consumer_offsets,29] on broker 1003: Shrinking ISR for partition [__consumer_offsets,29] from 1003,1002,1001 to 1003,1002 (kafka.cluster.Partition)
[2020-12-24 01:14:20,928] INFO Partition [__consumer_offsets,44] on broker 1003: Shrinking ISR for partition [__consumer_offsets,44] from 1003,1002,1001 to 1003,1002 (kafka.cluster.Partition)

Please suggest on this and Thanks in advance.

[#2]

@rohit_r_sharma 
Can you share the syntax for the topic creation? Is you cluster kerberized?  Is your Ranger Kafka plugin enabled? 
Please response and tag me!

[#3]

@Shelton ,

-Syntax of topic creation : 
kafka-topics.sh --create --topic "topic_name" --partitions 1 --replication-factor 2 --zookeeper broker_Fqdn1:2181,broker_Fqdn2:2181,broker_Fqdn3:2181

-Yes my cluster is kerberized.

-Yes ranger Kafka plugin is enabled

Thanks in advance.

[#4]

@rohit_sharma 

Can you change your syntax as below, note the zookeeper ensemble

/bin/kafka-topics.sh --create \
--zookeeper zk1:2181,zk2:2181,zk3:2181 \
--topic "topic_name" \
--partitions 1> \
--replication-factor 2

Hope that helps

[#5]

@rohit_sharma 
Any updates? Do you still need help?

[#6]

@Shelton ,
sorry for replying late, 

The syntax you gave is the same we used to create the topic, but one doubt is the ">" which is after 1 is typo error or the valid syntax. Also, we are using zk to create topics.

--partitions 1> \

 Still, i can see the error in Kafka:

 WARN [ReplicaFetcherThread-0-1001], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@11c4fb92 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 1001 was disconnected before the response was read

Thankyou for help in advance

[#7]

@rohit_sharma 

Indeed > was a typo

Can you share your code here ?