question Re: getkafka don't working in nifi in Archives of Support Questions (Read Only)

getkafka don't working in nifi

RAkhmadeev — Fri, 07 Oct 2016 21:49:58 GMT

When we trying to use getkafka we see following error:

2016-10-07 17:37:39,469 INFO [pool-24-thread-1-EventThread] org.I0Itec.zkclient.ZkClient zookeeper state changed (Expired)

2016-10-07 17:37:39,470 INFO [ZkClient-EventThread-465-hdp-name1.lab.croc.ru:2181] k.consumer.ZookeeperConsumerConnector [95446e62-0157-1000-7951-fd4244e9aec2_###############-1475841346967-f0d261ce], exception during rebalance

kafka.common.KafkaException: Failed to parse the broker info from zookeeper: {"jmx_port":-1,"timestamp":"1475501559373","endpoints":["PLAINTEXT://############:6667"],"host":"#############","version":3,"port":6667}

next we see:

Caused by: kafka.common.KafkaException: Unknown version of broker registration. Only versions 1 and 2 are supported.{"jmx_port":-1,"timestamp":"1475501559373","endpoints":["PLAINTEXT://#########:6667"],"host":"##########","version":3,"port":6667}

Our hdp version is 2.5 and hdf version is 2.0.

Re: getkafka don't working in nifi

MattWho — Fri, 07 Oct 2016 21:59:30 GMT

@Ramil Akhmadeev

HDP 2.5 comes with Apache Kafka 0.10.0.1. The NiFi getKafka processor uses the Kafka 0.8 client library. For communicating with Kafka 0.10 you should be using the consumeKafka_0_10 NiFi processor.

Re: getkafka don't working in nifi

RAkhmadeev — Fri, 07 Oct 2016 23:11:45 GMT

Thank You for quick reply, can You tell me please where can I get consumeKafka_o_10 nifi processor?

Re: getkafka don't working in nifi

RAkhmadeev — Fri, 07 Oct 2016 23:40:23 GMT

I've found, thank You very much for solution!

Re: getkafka don't working in nifi

MattWho — Fri, 07 Oct 2016 23:50:32 GMT

For other users/readers who do not know, HDF 2.0 includes as part of the release includes the following:

GetKafka and PutKafka --> Support Kafka 0.8

ConsumeKafka and PublishKafka --> Supports Kafka 0.9

ConsumeKafka_0_10 and PublishKafka_0_10 --> Supports Kafka 0.10

Thanks,

Matt