nifi 0.6.1 getkafka processor error: cannot connect to kafka

[#1]

ERROR 3491c89e-35c2-4443-a6b8-03303193629a

GetKafka[id=3491c89e-35c2-4443-a6b8-03303193629a] GetKafka[id=3491c89e-35c2-4443-a6b8-03303193629a] failed to process session due to java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Failed to retrieve partitions for topic test: java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Failed to retrieve partitions for topic test

Please let me know if i need to more properties rather than just the default properties in getKafka processor. Please let me know. thanks

[#2]

Can you describe what version of Kafka you're running. Have you been able to publish or consume messages from this topic?

[#3]

@jwitt iam using kafka 0.10.0.0. Just wondering the GetKafka processor doest have kafka broker config property how does it connect to kafka. It has zookeper property and topic to connect to. Any more properties i need to add. With just the default properties iam unable to connect to kafka running locally. thanks.

[#4 · Accepted solution]

Hello @SparkRocks , I am running Kafka 0.10.0.0 and successfully connected NiFi 0.6.1 GetKafka processor to it and consumed messages. All I had to do was specifying ZooKeeper Connection String and Topic Name.

As described in Kafka docs, Kafka 0.10.0.0 supports old consumer that depends on Zookeeper which doesn't have brokers configuration. NiFi GetKafka uses Kafka library ver 0.8.2.2, so it should work with 0.10.0.0.

Since I was running Zookeeper and Kafka on Docker instance, I set Zookeeper Connection String as "docker-host-address:2181".

Can you try to find where Zookeeper is running on, and NiFi has access to Zk? An easy way to test would be executing following command from where NiFi is running:

telnet zookeeper-host zookeeper-port

ex)$ telnet 192.168.99.100 2181
Trying 192.168.99.100...
Connected to cb675348f5c8.
Escape character is '^]'.
(Ctrl C to disconnect)

If it's not accessible, a msg like 'Connection refused, Unable to connect to remote host' will be returned instead. Hope this helps.

Looking at the other question you posted, it seems you've already got GetKafka connected successfully. Congratulations! If possible, please share what configuration was required for future reference 🙂

[#5]

hi @kkawamura i have used the same parameters zookeeper url & port and also the topic name. I thought it was about kafka versions but my issue was network connection issues with zookeeper. But its resolved now. Thank you 🙂 However as you looked at the other question just wondering any suggestions on connecting to S3 from as iam getting permission issues as described in the other question. Thank you