question Re: After upgrading from HDP 2.2 to HDP 2.3, all kafka topics are unaccessible by producers/consumers. in Archives of Support Questions (Read Only)

After upgrading from HDP 2.2 to HDP 2.3, all kafka topics are unaccessible by producers/consumers.

muanang — Wed, 02 Mar 2016 09:05:32 GMT

We upgraded our cluster from HDP 2.2.0 to HDP 2.3.

After the upgrade, we check the topics and all topics are still there.

But when trying to produce or even consume messages on all of the topics, we're having errors.

What should I do to fix my issue? I also notice on broker side that the broker.id also changed form id=0 to id=1001.

Please help.

nsabharwal — Wed, 02 Mar 2016 09:08:31 GMT

Try this ...Make sure that broker and zk is up. I have experienced the same issue and following helped me a lot

./kafka-console-producer.sh --broker-list kafkabrokerhost:6667 --topic test

./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

muanang — Wed, 02 Mar 2016 09:21:46 GMT

All services are up, including zookeeper and kafka (and brokers too).

I can see all the topics.

When trying to produce and consume messages using kafka tools, it's giving me errors .LeaderNotAvailableException].

This happens to all topics created under HDP 2.2 - Kafka 0.8.1.

But when I created a new topic, and try to produce and consume messages, ALL is WORKING.

I also noticed, the old topics are still saved in broker=0 while the new created ones are saved in broker=1001.

nsabharwal — Wed, 02 Mar 2016 09:24:26 GMT

@Michael Dennis Uanang And it's broken/mismatch after the upgrade?

muanang — Wed, 02 Mar 2016 09:32:19 GMT

Sorry but I didn't get what you mean.

Here's the flow.

Before upgrade:

HDP-2.2.0 | AMBARI-1.7 | KAFKA 0.8.1

After upgrade:

HDP-2.3.2 | AMBARI-2.2 | KAKFA 0.9.01 (on ambari)

nsabharwal — Wed, 02 Mar 2016 09:40:27 GMT

See this thread

If you have access to support then please open a ticket for deep troubleshooting.

muanang — Wed, 02 Mar 2016 09:47:37 GMT

Thanks @Neeraj Sabharwal for your response.

I was able to solve my own problem. 🙂

1. Stop Kafka on Ambari

2. SSH to kafka brokers and change the broker.id to it's original value. On my end I changed it from 1001,1002,1003 to 0,1,2 respectively.

3. Start Kafka on Ambari.

Cheers!