Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Creating a topic in kakfa fails with "ERROR kafka.admin.AdminOperationException: replication factor: 2 larger than available brokers: 0"

Solved Go to solution

Creating a topic in kakfa fails with "ERROR kafka.admin.AdminOperationException: replication factor: 2 larger than available brokers: 0"

Contributor
 HDP version : 2.5

Here are the steps which I have followed :

1. Installed kafka brokers on 2 nodes in my cluster.

2. Tried to create a topic , Created successfully

3. I tried to change the zookeeper root directory for kafka by following below steps :

> In ambari > Kafka > Configs > changed value of zookeeper.connect as follows :

node1.example.com:2181,node2.example.com:2181,node1.example.com:2181/kafka

4. Saved the changes and manually created /kafka in zookeeper using zkCli.sh

5. Restarted kafka brokers. After restart znodes were created in /kafka in zookeeper.

6. Now when I try to create a topic it fails with below stack trace :

Error while executing topic command : replication factor: 2 larger than available brokers: 0
[2016-12-02 16:13:12,851] ERROR kafka.admin.AdminOperationException: replication factor: 2 larger than available brokers: 0
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:117)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:403)
at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:110)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:61)
at kafka.admin.TopicCommand.main(TopicCommand.scala)

I can confirm that all znodes are created correctly under /kafka, brokers are up and running fine. I suspect that for some reason kafka admin or kafka controller(not sure who looks at the brokers znode value) is not able to identify live brokers because it tries to find the broker ids in previous zookeeper root dir i.e /brokers/ids/ which is empty.

In order to confirm if this is the scenario, I reverted the changes which were made.

 > Changed the zookeeper root directory to default i.e <node1.example.com:2181,node2.example.com:2181,node1.example.com:2181>

> Restarted zookeeper broker

> Confirmed that all znodes are with correct value (/controller_epoch) (/brokers/ids/1004 /brokers/ids/1006)

Now I tried to create a topic which got created without any issues.

Then I deleted both the broker IDS in /brokers/ids and tried to create a topic. It failed with same error i,e :

Error while executing topic command : replication factor: 2 larger than available brokers: 0 [2016-12-02 16:13:12,851] ERROR

Can some one please help me to understand this behaviour ?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Creating a topic in kakfa fails with "ERROR kafka.admin.AdminOperationException: replication factor: 2 larger than available brokers: 0"

@amankumbare can you make sure you are passing the zookeeper root as well to your kafka-topics.sh. It should look like this

./bin/kafka-topics.sh --zookeeper

  1. node1.example.com:2181,node2.example.com:2181,node1.example.com:2181/kafka --topic name_of_topic --partitions 2 --replication-factor 2 --create
3 REPLIES 3
Highlighted

Re: Creating a topic in kakfa fails with "ERROR kafka.admin.AdminOperationException: replication factor: 2 larger than available brokers: 0"

Contributor

Re: Creating a topic in kakfa fails with "ERROR kafka.admin.AdminOperationException: replication factor: 2 larger than available brokers: 0"

@amankumbare can you make sure you are passing the zookeeper root as well to your kafka-topics.sh. It should look like this

./bin/kafka-topics.sh --zookeeper

  1. node1.example.com:2181,node2.example.com:2181,node1.example.com:2181/kafka --topic name_of_topic --partitions 2 --replication-factor 2 --create

Re: Creating a topic in kakfa fails with "ERROR kafka.admin.AdminOperationException: replication factor: 2 larger than available brokers: 0"

Contributor
@Sriharsha Chintalapani

Yes ! It worked. Thanks a lot.

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1.openstacklocal:2181/kafka  --create  --topic my-topic  --partitions 2  --replication-factor 2 
Created topic "my-topic".
Don't have an account?
Coming from Hortonworks? Activate your account here