Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

avatar
Explorer
 
1 ACCEPTED SOLUTION

avatar
Contributor

To add some color to Bryan's response:

Kafka (Get/Put) can work with

  • Apache Kafka 0.8.x and 0.9.x broker in non-kerberized mode, without SSL
  • HDP 2.3 and 2.4 Kafka broker in both non-kerberized and kerberized modes, without SSL

Kafka (Consume/Publish) can work with

  • Apache Kafka 0.9.x and 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL
  • HDP 2.4 Kafka 0.9.x broker and HDP 2.5 Kafka 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL

View solution in original post

5 REPLIES 5

avatar
Master Guru

ConsumeKafka and PublishKafka use the Kafka 0.9 client library.

GetKafka and PutKafka use the Kafka 0.8 client library.

avatar
Contributor

To add some color to Bryan's response:

Kafka (Get/Put) can work with

  • Apache Kafka 0.8.x and 0.9.x broker in non-kerberized mode, without SSL
  • HDP 2.3 and 2.4 Kafka broker in both non-kerberized and kerberized modes, without SSL

Kafka (Consume/Publish) can work with

  • Apache Kafka 0.9.x and 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL
  • HDP 2.4 Kafka 0.9.x broker and HDP 2.5 Kafka 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL

avatar
Explorer

Thanks for a quick response. Also i see the consumekafka is connecting to broker instead of zookeeper, does it mean that it is doing a direct connect to kafka and not maintain the offset with zookeeper. Also where are the offsets stored in this case?

avatar
Contributor

ConsumeKafka is one of those processors whose states are stored and managed externally, outside of NIFI. Offsets are therefore maintained by kafka, instead of zookeeper.

Having said that, there is a plan to enhance state management for these processors. For more information, please refer to this JIRA ticket: https://issues.apache.org/jira/browse/NIFI-2078

avatar
Master Guru

Just to clarify, what Haimo mentioned is that the ConsumerKafka processor does not use any of NiFi's state management capabilities because the Kafka client maintains the offsets.

Regarding the Kafka client... in 0.9.0 I believe it no longer stores offsets in ZooKeeper, and now stores them internally somehow, so that is why you see it connecting directly to the broker and not using ZooKeeper.