all nodemanagers fail to start with error javax.security.auth.login.LoginException: Unable to obtain password from user

[#1]

I see the following error in NodeManager startup log

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed NodeManager login
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:300)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:547)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)
Caused by: java.io.IOException: Login failure for nm/node28.my.com@HDP from keytab nm.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user


        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1098)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:307)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:271)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.doSecureLogin(NodeManager.java:156)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:298)
        ... 3 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user


        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1089)
        ... 7 more

but when I do

kinit -kt /etc/security/keytabs/nm.service.keytab nm/node28.my.com@HDP

and then

klist

Ticket cache: FILE:/tmp/krb5cc_1011
Default principal: nm/node28.my.com@HDP


Valid starting       Expires              Service principal
06/21/2017 18:30:06  06/22/2017 18:30:06  krbtgt/HDP@HDP

Unable to figure out why Nodemanager fails with that exception!

[#2]

Turned out to be a silly mistake - this line had the clue - Login failure for nm/node28.my.com@HDP from keytab nm.service.keytab. The keytab's absolute path should be in the message. In ambari blueprint, node manager keytab configuration did not have the absolute path which is - /etc/security/keytabs/nm.service.keytab