Nifi Version 1.1.0.2.1.0.0-165 1 - Cannot get GetKafka connector to connect to Kafka - Single server - No Keberos - Zookeeper Port confirmed

[#1]

I am unable to get the GetKafka connector to connect to Kafka.

I have a single server install of HDF-2.1. NiFi, Kafka, and ZooKeeper are all installed on the same server. There is no security. I have verified the ZooKeeper port by connecting to ZooKeeper from other machines in this manner:

ex)$ telnet 192.168.99.100 2181 Trying 192.168.99.100... Connected to cb675348f5c8. Escape character is'^]' (Ctrl C to disconnect)

Here is the error I get from the GetKafka connector:

Any help would be greatly appreciated.

[#2 · Accepted solution]

What version of the Kafka broker are you running?

If its installed through HDF, then it is probably Kafka 0.10.x and you should try using ConsumeKafka_0_10.

In general it is best to use the processor that goes with the broker version, so..

GetKakfa - Kafka 0.8

ConsumeKafka - Kafka 0.9

ConsumeKafka_0_10 - Kafka 0.1.0

[#3]

I'm not sure how the to identify the Kafka broker version easily. I did find the the file kafka_2.10-0.10.1.2.1.0.0-165.jar.asc in the /libs folder where Kafka is installed so I am assuming I am running Kafka 0.10.1.

I did get both the ConsumeKafka and ConsumeKafka_0_10 connectors to work.

Thanks.

Now off to figure out why the PutHiveStreaming doesn't work, but that will be for a different post.