NiFi Consume Kafka 010 not ingesting messages but Producer and Consumer work

[#1]

Hi,

I feel like this question has been asked before in similar ways but each response has yet to yield a result that has yet to work for me. So, I have an Amazon Linux machine with Apache Kafka 0.10.0 on it and when I start zookeeper and kafka I am able to successfully produce and consume messages on that machine.

On a separate Amazon Windows server I have Apache Nifi 1.6 installed and I am using the Consume Kafka 0_10 processor to try and receive messages from the topic that was created on my standalone kafka install (Linux machine). I have tried to use the ip address of the Linux machine as the broker within the NiFi processor with port 9092 as well as the Ec2 instance name ec2-XX-X-XXX-XXX.XX-XXXX-1.compute.amazonaws.com:9092 but neither show any data coming in when I manually type in messages from the Kafka producer on the Linux machine. There are no errors produced in the logs of NiFi and I I know that port 9092 is open so I would think the messages would come through.

In the server.properties file for Kafka I have the listeners set to listeners=PLAINTEXT://:9092 (bases on some stack overflow post recommendation). In that same file I also have advertised.listeners=PLAINTEXT://XX-X-XXX-XXX.XX-XXXX-1.compute.amazonaws.com:9092.

I have also tried NiFi 1.5 and to see if this was a version issue but nothing seems to work. I would think that if Nifi is not producing errors, the proper ports are open and I can produce and consume locally to Linux machine then I would think there should be no reason the Consume Kafka 0_10 couldn't pick up the messages I type and parse them into a flow file.

Sorry for the long question but if anyone has some insight on this issue please let me know.

Thanks, Jason

[#2]

@Vinicius Higa Murakami - Thanks for the suggestion. I looked at the post you sent and tried to access some of the steps but it seems that I don't have the zk.cli.sh file that was suggested. I am running zookeeper as part of a Kafka download and it does not have this file. We have a current project that is utilizing Kafka and I decided to try connecting from there as well but no luck either.

Jason

[#3]

Hey @Jason Sphar!
Did you tried to use zookeeper-cli instead of zk-cli.sh ?

[#4]

@Jason Sphar

What about the windows firewall? Try turning it off completely and see if it works then.

[#5]

@Wynner

Thanks for the suggestion, I have not tried this yet but will have to check with the server administrator to see if this is something they don't mind trying.

Jason