question Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka in Archives of Support Questions (Read Only)

Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

anand140778 — Wed, 03 Aug 2016 22:55:25 GMT

Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

bbende — Wed, 03 Aug 2016 23:10:25 GMT

ConsumeKafka and PublishKafka use the Kafka 0.9 client library.

GetKafka and PutKafka use the Kafka 0.8 client library.

Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

hliu — Wed, 03 Aug 2016 23:29:45 GMT

To add some color to Bryan's response:

Kafka (Get/Put) can work with

Apache Kafka 0.8.x and 0.9.x broker in non-kerberized mode, without SSL
HDP 2.3 and 2.4 Kafka broker in both non-kerberized and kerberized modes, without SSL

Kafka (Consume/Publish) can work with

Apache Kafka 0.9.x and 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL
HDP 2.4 Kafka 0.9.x broker and HDP 2.5 Kafka 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL

Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

anand140778 — Thu, 04 Aug 2016 03:28:58 GMT

Thanks for a quick response. Also i see the consumekafka is connecting to broker instead of zookeeper, does it mean that it is doing a direct connect to kafka and not maintain the offset with zookeeper. Also where are the offsets stored in this case?

Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

hliu — Thu, 04 Aug 2016 03:46:12 GMT

ConsumeKafka is one of those processors whose states are stored and managed externally, outside of NIFI. Offsets are therefore maintained by kafka, instead of zookeeper.

Having said that, there is a plan to enhance state management for these processors. For more information, please refer to this JIRA ticket: https://issues.apache.org/jira/browse/NIFI-2078

Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

bbende — Thu, 04 Aug 2016 03:59:41 GMT

Just to clarify, what Haimo mentioned is that the ConsumerKafka processor does not use any of NiFi's state management capabilities because the Kafka client maintains the offsets.

Regarding the Kafka client... in 0.9.0 I believe it no longer stores offsets in ZooKeeper, and now stores them internally somehow, so that is why you see it connecting directly to the broker and not using ZooKeeper.