- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
NiFi unable to connect to kafka in kerberized cluster
- Labels:
-
Apache Kafka
-
Apache NiFi
Created ‎04-17-2017 09:41 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello all, we are having issues connecting from NiFi (1.1) to Kafka (0.9) server using ConsumeKafka processor on kerberos enabled cluster. We were able to execute the same process in one environment (dev), but getting below error in test environment.
"Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. Make sure -Djava.security.auth.login.config property passed to JVM and the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)'. Make sure you are using FQDN of the Kafka broker you are trying to connect to. not available to garner authentication information from the user"
we have updated the bootstrap.conf file to use the correct jaas.conf file
java.arg.15=-Djava.security.auth.login.config=/data/configuration_resources/jaas.conf
updated jaas.conf file to use correct principal and keytab:
KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/data/configuration_resources/kafka.keytab" principal="kafka/kafka1.hostname.com@EXAMPLE.COM"; };
and updated client.properties file
security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name=kafka
added krb5.conf info in the nifi.properties file
nifi.kerberos.krb5.file=/data/configuration_resources/krb5.conf
what are the other files/configurations that we have to check to resolve this issue?
Created ‎04-18-2017 01:45 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Add the line "useTicketCache=true" to the jaas.conf file.
What are the differences between the dev and test environments? Are you using the same KDC server? Same kafka brokers?
Created ‎04-18-2017 12:24 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you run the klist command as the user running NiFi, what do you see?
Created ‎04-18-2017 12:57 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@wynner we use three different users. one user (ser_nifi) to start/run the nifi service, "nifi" user to connect to hdfs and "kafka" user to connect to kafka. we were able to do kinit using "kafka" user without an issue.
in this same env, we were able to connect to hbase on the same server and write data into a table. only kafka connector is throwing this error.
Created ‎04-18-2017 01:45 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Add the line "useTicketCache=true" to the jaas.conf file.
What are the differences between the dev and test environments? Are you using the same KDC server? Same kafka brokers?
Created ‎04-20-2017 01:44 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am glad you were able to correct the issue by configuring the correct KDC server in the jaas.conf file.
Created ‎04-19-2017 01:27 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @wynner, it was indeed an issue with the incorrect KDC server specified in the jaas.conf file. changing that fixed the error. thanks for looking into this
