Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

After upgrading from HDP 2.2 to HDP 2.3, all kafka topics are unaccessible by producers/consumers.

Solved Go to solution

After upgrading from HDP 2.2 to HDP 2.3, all kafka topics are unaccessible by producers/consumers.

Expert Contributor

We upgraded our cluster from HDP 2.2.0 to HDP 2.3.

After the upgrade, we check the topics and all topics are still there.

But when trying to produce or even consume messages on all of the topics, we're having errors.

What should I do to fix my issue? I also notice on broker side that the broker.id also changed form id=0 to id=1001.

Please help.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: After upgrading from HDP 2.2 to HDP 2.3, all kafka topics are unaccessible by producers/consumers.

Expert Contributor

Thanks @Neeraj Sabharwal for your response.

I was able to solve my own problem. :)

1. Stop Kafka on Ambari

2. SSH to kafka brokers and change the broker.id to it's original value. On my end I changed it from 1001,1002,1003 to 0,1,2 respectively.

3. Start Kafka on Ambari.

Cheers!

View solution in original post

6 REPLIES 6
Highlighted

Re: After upgrading from HDP 2.2 to HDP 2.3, all kafka topics are unaccessible by producers/consumers.

@Michael Dennis Uanang

Try this ...Make sure that broker and zk is up. I have experienced the same issue and following helped me a lot

./kafka-console-producer.sh --broker-list kafkabrokerhost:6667 --topic test

./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Highlighted

Re: After upgrading from HDP 2.2 to HDP 2.3, all kafka topics are unaccessible by producers/consumers.

Expert Contributor

Hi @Neeraj Sabharwal,

All services are up, including zookeeper and kafka (and brokers too).

I can see all the topics.

When trying to produce and consume messages using kafka tools, it's giving me errors .LeaderNotAvailableException].

This happens to all topics created under HDP 2.2 - Kafka 0.8.1.

But when I created a new topic, and try to produce and consume messages, ALL is WORKING.

I also noticed, the old topics are still saved in broker=0 while the new created ones are saved in broker=1001.

Highlighted

Re: After upgrading from HDP 2.2 to HDP 2.3, all kafka topics are unaccessible by producers/consumers.

@Michael Dennis Uanang And it's broken/mismatch after the upgrade?

Highlighted

Re: After upgrading from HDP 2.2 to HDP 2.3, all kafka topics are unaccessible by producers/consumers.

Expert Contributor
@Neeraj

Sorry but I didn't get what you mean.

Here's the flow.

Before upgrade:

HDP-2.2.0 | AMBARI-1.7 | KAFKA 0.8.1

After upgrade:

HDP-2.3.2 | AMBARI-2.2 | KAKFA 0.9.01 (on ambari)

Highlighted

Re: After upgrading from HDP 2.2 to HDP 2.3, all kafka topics are unaccessible by producers/consumers.

@Michael Dennis Uanang

See this thread

If you have access to support then please open a ticket for deep troubleshooting.

Highlighted

Re: After upgrading from HDP 2.2 to HDP 2.3, all kafka topics are unaccessible by producers/consumers.

Expert Contributor

Thanks @Neeraj Sabharwal for your response.

I was able to solve my own problem. :)

1. Stop Kafka on Ambari

2. SSH to kafka brokers and change the broker.id to it's original value. On my end I changed it from 1001,1002,1003 to 0,1,2 respectively.

3. Start Kafka on Ambari.

Cheers!

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here