NiFi PutHDFS Login Failure using kerberos

[#1]

I am getting a login failure when trying to use the PutHDFS processor.

These are the things I have verified:

• nifi.kerberos.krb5.file is set correctly
• nifi account has read permissions to keytab and krb5.conf
• I am able to kinit using the keytab/principal from the nifi account

Environment is HDF 3.0.1 (NiFi 1.2) to HDP 2.6.x

Is there something I am missing?

[#2]

@Benjamin Hopp

Did you copy over the core-site.xml and hdfs-site.xml files from the HDP cluster after enabling kerberos?

[#3]

Yes, and verified permissions on those files as well.

[#4]

@Benjamin Hopp

How is the PutHDFS processor configured?

[#5]

Other than the config xmls and kerberos everything is default.

[#6]

@Benjamin Hopp

Is this a cluster or standalone instance?

Also, would you post to complete error stack from the nifi-app.log file?

[#7]

Both HDP and HDF are clustered.

2017-09-29 16:42:41,239 ERROR [StandardProcessScheduler Thread-6] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
        at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:102)
        at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
        ... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)
        at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1305)
        at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1301)
        ... 6 common frames omitted
Caused by: java.io.IOException: Login failure for NiFi_App_Prod from keytab /etc/security/keytabs/NiFi_App_Prod.keytab
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1183)
        at org.apache.nifi.hadoop.SecurityUtil.loginKerberos(SecurityUtil.java:54)
        at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:271)
        at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:204)
        at org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:191)
        ... 16 common frames omitted
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user


        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1172)
        ... 20 common frames omitted

[#8]

@Benjamin Hopp

You need to use a headless keytab, so it does not require a password. There is no way to provide the password in the processor.

[#9]

The existing keytab should not require a password. The following command works, and doesn't prompt for a password.

kinit -kt /etc/security/keytabs/NiFi_App_Prod.keytab NiFi_App_Prod@<REALM>.com 

Also, a similar configuration works in the dev environment without issue.

[#10]

@Benjamin Hopp

Can you try to validate by doing the below steps

$ kdestroy 

The grab a ticket as the nifi user

[root@host ~]# sudo su - nifi_user
$ kinit
Password for nifi_user@NAM.xxxxxxxx.COM: 
$ klist
Ticket cache: FILE:/tmp/krb5cc_49393
Default principal: nifi_user@NAM.xxxxxxxx.COM
Valid starting     Expires            Service principal
09/28/17 17:06:44  09/29/17 03:06:44  krbtgt/ NAM.xxxxxxxx.COM@NAM.xxxxxxxx.COM

Tell me if that works

[#11]

@Benjamin Hopp

Did you get this resolved? I'm facing the exact same issue as you are in this thread.

[#12]

@Chad Woodhead

Yes, it was resolved. But I don't have a good explanation. It actually resolved itself by simply restarting the NiFi process on all nodes. After the restart they all connected and worked as expected.

[#13]

@Benjamin Hopp

Interesting. I performed several restarts and kept receiving the error "Unable to obtain password from user". I then decided to perform a full stop, wait a couple minutes, then start (I have 2 NiFi nodes) and now its working. Very strange...

[#14]

@Benjamin Hopp @Chad Woodhead

I had the exact same issue and tried bringing down both of my two NiFi nodes, waiting a few minutes, and brought them back online. Then I tried turning on the PutHDFS processor and it worked properly.

Has anyone figured out why this solves the issue or what is causing this problem?