Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

GetKafka: failing to read data from Kerbaros enabled kafka

GetKafka: failing to read data from Kerbaros enabled kafka

Hi,

I am trying to read the data from kafka topic on the cluster where kerbaros is enabled. I am getting errors like below for all the topics that I am trying to read the data from.

***************Error logs*******************

2017-08-17 06:18:26,505 INFO [test-consumer-group_storagev4-1c-1502965058303-e4ac2072-leader-finder-thread] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1502965058340] Added fetcher for partitions ArrayBuffer() 2017-08-17 06:18:26,599 WARN [test-consumer-group_storagev4-1c-1502965084565-3028bb7a-leader-finder-thread] k.c.ConsumerFetcherManager$LeaderFinderThread [test-consumer-group_storagev4-1c-1502965084565-3028bb7a-leader-finder-thread], Failed to find leader for Set([demo_unix_indexing,0]) java.lang.NullPointerException: null 2017-08-17 06:18:26,599 INFO [test-consumer-group_storagev4-1c-1502965084565-3028bb7a-leader-finder-thread] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1502965084750] Added fetcher for partitions ArrayBuffer() 2017-08-17 06:18:26,710 WARN [test-consumer-group_storagev4-1c-1502965058303-e4ac2072-leader-finder-thread] k.c.ConsumerFetcherManager$LeaderFinderThread [test-consumer-group_storagev4-1c-1502965058303-e4ac2072-leader-finder-thread], Failed to find leader for Set([indexing,0]) java.lang.NullPointerException: null 2017-08-17 06:18:26,710 INFO [test-consumer-group_storagev4-1c-1502965058303-e4ac2072-leader-finder-thread] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1502965058340] Added fetcher for partitions ArrayBuffer() 2017-08-17 06:18:26,805 WARN [test-consumer-group_storagev4-1c-1502965084565-3028bb7a-leader-finder-thread] k.c.ConsumerFetcherManager$LeaderFinderThread [test-consumer-group_storagev4-1c-1502965084565-3028bb7a-leader-finder-thread], Failed to find leader for Set([demo_unix_indexing,0]) java.lang.NullPointerException: null

********************

I have done the necessary configurations but I could not able to figure out the cause of the issue. Note: I am able to read the data from kafka topic with consumeKafka processor.

Nifi details: Nifi is running on standalone single node. Zookeeper and kafka are running on 3 node cluster.

***********jass file configiuration*******

Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/nifi.keytab" storeKey=true useTicketCache=false principal="nifi@TESTSSTECH.LOCAL"; };

KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="kafka" useKeyTab=true keyTab="/etc/security/keytabs/nifi.keytab" principal="nifi@TESTSSTECH.LOCAL"; };

**************************

6 REPLIES 6

Re: GetKafka: failing to read data from Kerbaros enabled kafka

If you can consume the data with ConsumeKafka then why do you want to use GetKafka?

GetKafka should be used with 0.8 Kafka broker

ConsumeKafka should be used with the 0.9 Kafka broker

ConsumeKafka_0_10 should be used with the 0.10 Kafka broker

Re: GetKafka: failing to read data from Kerbaros enabled kafka

Please see this post: https://community.hortonworks.com/questions/49365/apache-nifi-what-is-the-difference-between-consume...

The ConsumeKakfa processor uses the Kafka 0.9 client library and depending on the version of the brokers to which you are connecting, the 0.8 version of the library may not be compatible as discussed in that post.

I would just use ConsumeKafka if that's working for you.

Re: GetKafka: failing to read data from Kerbaros enabled kafka

when I use consume kafka, I am facing other issue as mentioned in the below link

https://community.hortonworks.com/answers/114429/view.html

Re: GetKafka: failing to read data from Kerbaros enabled kafka

That could be related to https://issues.apache.org/jira/browse/KAFKA-3627 which was resolved in 0.10 or could be a broker-level issue within your environment

Re: GetKafka: failing to read data from Kerbaros enabled kafka

I tried the possible settings with higher session timeouts. But I could not able to resolve the issue with consumeKafka. Where as I dint faced such issue with getKafka processor. So, I am using getKafka.

But now I am facing the issue with getKafka in kerberos enabled environment. I would to know how to configure getKafka in kerberos enabled environment and would like to hear possible workarounds to resolve the issue that I mentioned in the logs.

Re: GetKafka: failing to read data from Kerbaros enabled kafka

What version of NiFi are you using?

There was a previous issue where if the queue filled up between ConsumeKafka and the next processor, then backpressure was applied which caused ConsumeKafka to not execute for longer than the session timeout, which then caused a rebalance to occur.

This was resolved in Apache NiFi 1.2.0: https://issues.apache.org/jira/browse/NIFI-3189

In addition, you should try batching together many messages into a single flow file by setting Message Demarcator in ConsumeKafka. You will get much better performance overall, and you will put less flow files into the next queue and help avoid the above situation.

Don't have an account?
Coming from Hortonworks? Activate your account here