Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

GetKafka working for Kafka 0.10 but not ConsumeKafka or ConsumeKafka_0_10

avatar
Expert Contributor

13353-screen-shot-2017-03-07-at-14745-pm.png

13354-screen-shot-2017-03-07-at-14959-pm.png

For some reason even though my Kafka version is listed as kafka_2.10-0.10.0.2.5.0.0-1245.jar (which I am pretty sure is Kafka 0.10), the only processor that works for me is GetKafka.

Consume_Kafka and Consume_Kafka_0_10 both will hang and not process any data... if I stop the nifi flow it does give an error seen in the screenshot. I have left it running up to 10 mins waiting for it to connect but it never seems to.

I have also including a screenshot of my KafkaConsume properties window which is using the same values as the ones I put in GetKafka when it works.

1 ACCEPTED SOLUTION

avatar

Hi @Eric Lloyd,

Is your Kafka broker listening on port 2181? It looks like a zookeeper port.

As indicated in the properties description, ConsumeKafka processors are not using ZooKeeper to connect to the brokers as GetKafka is doing. If you provide a comma separated list of your brokers (with corresponding port), it should be OK.

Hope this helps.

View solution in original post

5 REPLIES 5

avatar

Hi @Eric Lloyd,

Is your Kafka broker listening on port 2181? It looks like a zookeeper port.

As indicated in the properties description, ConsumeKafka processors are not using ZooKeeper to connect to the brokers as GetKafka is doing. If you provide a comma separated list of your brokers (with corresponding port), it should be OK.

Hope this helps.

avatar
Expert Contributor

Crap. I thought the Zookeeper port and the Kafka brokers port were the same. Ill hunt down what ports my Kafka broker is listening on - not sure where to begin looking. Any suggestion? ( Im sure Ill find it through Google)

avatar

With plaintext, I believe default port should be 9092.

avatar
Expert Contributor

Got it! Thanks!

avatar
Contributor

Hi Eric, did you solved the issue?

I'm having the same problem when trying to use processors ConsumeKafka and ConsumeKafka_0_10 to read messages from kafka 0.10.1 (HDP 2.6 without kerberos). I did tests with different parameter settings and the processors starts without error but never receives the messages. I had same results executing the kafka-console-consumer shell with bootstrap-server parameter.

Can you help me? Any idea of what's its going on?