Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

Solved Go to solution
Highlighted

Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

Explorer
 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

Explorer

To add some color to Bryan's response:

Kafka (Get/Put) can work with

  • Apache Kafka 0.8.x and 0.9.x broker in non-kerberized mode, without SSL
  • HDP 2.3 and 2.4 Kafka broker in both non-kerberized and kerberized modes, without SSL

Kafka (Consume/Publish) can work with

  • Apache Kafka 0.9.x and 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL
  • HDP 2.4 Kafka 0.9.x broker and HDP 2.5 Kafka 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL

View solution in original post

5 REPLIES 5
Highlighted

Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

ConsumeKafka and PublishKafka use the Kafka 0.9 client library.

GetKafka and PutKafka use the Kafka 0.8 client library.

Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

Explorer

To add some color to Bryan's response:

Kafka (Get/Put) can work with

  • Apache Kafka 0.8.x and 0.9.x broker in non-kerberized mode, without SSL
  • HDP 2.3 and 2.4 Kafka broker in both non-kerberized and kerberized modes, without SSL

Kafka (Consume/Publish) can work with

  • Apache Kafka 0.9.x and 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL
  • HDP 2.4 Kafka 0.9.x broker and HDP 2.5 Kafka 0.10.x broker in both non-kerberized and kerberized modes, with and without SSL

View solution in original post

Highlighted

Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

Explorer

Thanks for a quick response. Also i see the consumekafka is connecting to broker instead of zookeeper, does it mean that it is doing a direct connect to kafka and not maintain the offset with zookeeper. Also where are the offsets stored in this case?

Highlighted

Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

Explorer

ConsumeKafka is one of those processors whose states are stored and managed externally, outside of NIFI. Offsets are therefore maintained by kafka, instead of zookeeper.

Having said that, there is a plan to enhance state management for these processors. For more information, please refer to this JIRA ticket: https://issues.apache.org/jira/browse/NIFI-2078

Highlighted

Re: Apache NIFI - What is the difference between ConsumeKafka and GetKafka,Apache Nifi - What is the difference between ConsumeKafka and GetKafka

Just to clarify, what Haimo mentioned is that the ConsumerKafka processor does not use any of NiFi's state management capabilities because the Kafka client maintains the offsets.

Regarding the Kafka client... in 0.9.0 I believe it no longer stores offsets in ZooKeeper, and now stores them internally somehow, so that is why you see it connecting directly to the broker and not using ZooKeeper.

Don't have an account?
Coming from Hortonworks? Activate your account here