Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi PutHDFS Login Failure using kerberos

NiFi PutHDFS Login Failure using kerberos

Rising Star

I am getting a login failure when trying to use the PutHDFS processor.

These are the things I have verified:

  • nifi.kerberos.krb5.file is set correctly
  • nifi account has read permissions to keytab and krb5.conf
  • I am able to kinit using the keytab/principal from the nifi account

Environment is HDF 3.0.1 (NiFi 1.2) to HDP 2.6.x

Is there something I am missing?

13 REPLIES 13

Re: NiFi PutHDFS Login Failure using kerberos

@Benjamin Hopp

Did you copy over the core-site.xml and hdfs-site.xml files from the HDP cluster after enabling kerberos?

Highlighted

Re: NiFi PutHDFS Login Failure using kerberos

Rising Star

Yes, and verified permissions on those files as well.

Re: NiFi PutHDFS Login Failure using kerberos

@Benjamin Hopp

How is the PutHDFS processor configured?

Re: NiFi PutHDFS Login Failure using kerberos

Rising Star

Other than the config xmls and kerberos everything is default.

39631-puthdfs.png

Re: NiFi PutHDFS Login Failure using kerberos

@Benjamin Hopp

Is this a cluster or standalone instance?

Also, would you post to complete error stack from the nifi-app.log file?

Re: NiFi PutHDFS Login Failure using kerberos

Rising Star

Both HDP and HDF are clustered.

2017-09-29 16:42:41,239 ERROR [StandardProcessScheduler Thread-6] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
        at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:102)
        at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
        ... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)
        at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)
        at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1305)
        at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1301)
        ... 6 common frames omitted
Caused by: java.io.IOException: Login failure for NiFi_App_Prod from keytab /etc/security/keytabs/NiFi_App_Prod.keytab
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1183)
        at org.apache.nifi.hadoop.SecurityUtil.loginKerberos(SecurityUtil.java:54)
        at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:271)
        at org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:204)
        at org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:191)
        ... 16 common frames omitted
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user


        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1172)
        ... 20 common frames omitted



Re: NiFi PutHDFS Login Failure using kerberos

@Benjamin Hopp

You need to use a headless keytab, so it does not require a password. There is no way to provide the password in the processor.

Re: NiFi PutHDFS Login Failure using kerberos

Rising Star

The existing keytab should not require a password. The following command works, and doesn't prompt for a password.

kinit -kt /etc/security/keytabs/NiFi_App_Prod.keytab NiFi_App_Prod@<REALM>.com 

Also, a similar configuration works in the dev environment without issue.

Re: NiFi PutHDFS Login Failure using kerberos

Mentor

@Benjamin Hopp

Can you try to validate by doing the below steps

$ kdestroy 

The grab a ticket as the nifi user

[root@host ~]# sudo su - nifi_user
$ kinit
Password for nifi_user@NAM.xxxxxxxx.COM: 
$ klist
Ticket cache: FILE:/tmp/krb5cc_49393
Default principal: nifi_user@NAM.xxxxxxxx.COM
Valid starting     Expires            Service principal
09/28/17 17:06:44  09/29/17 03:06:44  krbtgt/ NAM.xxxxxxxx.COM@NAM.xxxxxxxx.COM

Tell me if that works