question Re: NiFi unable to connect to kafka in kerberized cluster in Archives of Support Questions (Read Only)

NiFi unable to connect to kafka in kerberized cluster

harishkumar_swa — Tue, 18 Apr 2017 04:41:47 GMT

Hello all, we are having issues connecting from NiFi (1.1) to Kafka (0.9) server using ConsumeKafka processor on kerberos enabled cluster. We were able to execute the same process in one environment (dev), but getting below error in test environment.

"Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. Make sure -Djava.security.auth.login.config property passed to JVM and the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)'. Make sure you are using FQDN of the Kafka broker you are trying to connect to. not available to garner authentication information from the user"

we have updated the bootstrap.conf file to use the correct jaas.conf file

java.arg.15=-Djava.security.auth.login.config=/data/configuration_resources/jaas.conf

updated jaas.conf file to use correct principal and keytab:

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/data/configuration_resources/kafka.keytab"
principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};

and updated client.properties file

security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka

added krb5.conf info in the nifi.properties file

nifi.kerberos.krb5.file=/data/configuration_resources/krb5.conf

what are the other files/configurations that we have to check to resolve this issue?

Re: NiFi unable to connect to kafka in kerberized cluster

Wynner — Tue, 18 Apr 2017 19:24:46 GMT

@Harish S

If you run the klist command as the user running NiFi, what do you see?

Re: NiFi unable to connect to kafka in kerberized cluster

harishkumar_swa — Tue, 18 Apr 2017 19:57:32 GMT

@wynner we use three different users. one user (ser_nifi) to start/run the nifi service, "nifi" user to connect to hdfs and "kafka" user to connect to kafka. we were able to do kinit using "kafka" user without an issue.

in this same env, we were able to connect to hbase on the same server and write data into a table. only kafka connector is throwing this error.

Re: NiFi unable to connect to kafka in kerberized cluster

Wynner — Tue, 18 Apr 2017 20:45:49 GMT

@Harish S

Add the line "useTicketCache=true" to the jaas.conf file.

What are the differences between the dev and test environments? Are you using the same KDC server? Same kafka brokers?

Re: NiFi unable to connect to kafka in kerberized cluster

harishkumar_swa — Wed, 19 Apr 2017 08:27:20 GMT

Hi @wynner, it was indeed an issue with the incorrect KDC server specified in the jaas.conf file. changing that fixed the error. thanks for looking into this

Re: NiFi unable to connect to kafka in kerberized cluster

Wynner — Thu, 20 Apr 2017 20:44:33 GMT

@Harish S

I am glad you were able to correct the issue by configuring the correct KDC server in the jaas.conf file.