Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

GetKafka working for Kafka 0.10 but not ConsumeKafka or ConsumeKafka_0_10

SOLVED Go to solution
Highlighted

GetKafka working for Kafka 0.10 but not ConsumeKafka or ConsumeKafka_0_10

Rising Star

13353-screen-shot-2017-03-07-at-14745-pm.png

13354-screen-shot-2017-03-07-at-14959-pm.png

For some reason even though my Kafka version is listed as kafka_2.10-0.10.0.2.5.0.0-1245.jar (which I am pretty sure is Kafka 0.10), the only processor that works for me is GetKafka.

Consume_Kafka and Consume_Kafka_0_10 both will hang and not process any data... if I stop the nifi flow it does give an error seen in the screenshot. I have left it running up to 10 mins waiting for it to connect but it never seems to.

I have also including a screenshot of my KafkaConsume properties window which is using the same values as the ones I put in GetKafka when it works.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: GetKafka working for Kafka 0.10 but not ConsumeKafka or ConsumeKafka_0_10

Hi @Eric Lloyd,

Is your Kafka broker listening on port 2181? It looks like a zookeeper port.

As indicated in the properties description, ConsumeKafka processors are not using ZooKeeper to connect to the brokers as GetKafka is doing. If you provide a comma separated list of your brokers (with corresponding port), it should be OK.

Hope this helps.

5 REPLIES 5

Re: GetKafka working for Kafka 0.10 but not ConsumeKafka or ConsumeKafka_0_10

Hi @Eric Lloyd,

Is your Kafka broker listening on port 2181? It looks like a zookeeper port.

As indicated in the properties description, ConsumeKafka processors are not using ZooKeeper to connect to the brokers as GetKafka is doing. If you provide a comma separated list of your brokers (with corresponding port), it should be OK.

Hope this helps.

Re: GetKafka working for Kafka 0.10 but not ConsumeKafka or ConsumeKafka_0_10

Rising Star

Crap. I thought the Zookeeper port and the Kafka brokers port were the same. Ill hunt down what ports my Kafka broker is listening on - not sure where to begin looking. Any suggestion? ( Im sure Ill find it through Google)

Re: GetKafka working for Kafka 0.10 but not ConsumeKafka or ConsumeKafka_0_10

With plaintext, I believe default port should be 9092.

Re: GetKafka working for Kafka 0.10 but not ConsumeKafka or ConsumeKafka_0_10

Rising Star

Got it! Thanks!

Re: GetKafka working for Kafka 0.10 but not ConsumeKafka or ConsumeKafka_0_10

New Contributor

Hi Eric, did you solved the issue?

I'm having the same problem when trying to use processors ConsumeKafka and ConsumeKafka_0_10 to read messages from kafka 0.10.1 (HDP 2.6 without kerberos). I did tests with different parameter settings and the processors starts without error but never receives the messages. I had same results executing the kafka-console-consumer shell with bootstrap-server parameter.

Can you help me? Any idea of what's its going on?