Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

[#1]

ConsumeKafka and PublishKafka use the Kafka 0.9 client library.

GetKafka and PutKafka use the Kafka 0.8 client library.

[#2 · Accepted solution]

To add some color to Bryan's response:

Kafka (Get/Put) can work with

• Apache Kafka 0.8.x and 0.9.x broker in non-kerberized mode, without SSL
• HDP 2.3 and 2.4 Kafka broker in both non-kerberized and kerberized modes, without SSL

Kafka (Consume/Publish) can work with

• Apache Kafka 0.9.x and 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL
• HDP 2.4 Kafka 0.9.x broker and HDP 2.5 Kafka 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL

[#3]

Thanks for a quick response. Also i see the consumekafka is connecting to broker instead of zookeeper, does it mean that it is doing a direct connect to kafka and not maintain the offset with zookeeper. Also where are the offsets stored in this case?

[#4]

ConsumeKafka is one of those processors whose states are stored and managed externally, outside of NIFI. Offsets are therefore maintained by kafka, instead of zookeeper.

Having said that, there is a plan to enhance state management for these processors. For more information, please refer to this JIRA ticket: https://issues.apache.org/jira/browse/NIFI-2078

[#5]

Just to clarify, what Haimo mentioned is that the ConsumerKafka processor does not use any of NiFi's state management capabilities because the Kafka client maintains the offsets.

Regarding the Kafka client... in 0.9.0 I believe it no longer stores offsets in ZooKeeper, and now stores them internally somehow, so that is why you see it connecting directly to the broker and not using ZooKeeper.