Failure to login: for principal NodeManager From keytab

[#1]

Hi Everyone can help me,

I'm strat NodeManger in ambari but show error "failure to login: for principal: nm/[email protected] from keytab /etc/security/keytabs/nm.service.keytab"  for detail like below

2024-06-11 09:30:28,202 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(611)) - NodeManager metrics system shutdown complete.
2024-06-11 09:30:28,202 ERROR nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(965)) - Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed NodeManager login
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:488)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:962)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1042)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: nm/[email protected] from keytab /etc/security/keytabs/nm.service.keytab javax.security.auth.login.LoginException: Unable to obtain password from user

	at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2012)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1365)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1125)
	at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:324)
	at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:288)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.doSecureLogin(NodeManager.java:295)
	at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:486)
	... 3 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

	at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:903)
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:766)
	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2091)
	at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2001)
	... 9 more
2024-06-11 09:30:28,204 INFO  nodemanager.NodeManager (LogAdapter.java:info(51)) - SHUTDOWN_MSG: 

any suggestions?

[#2]

Hello @rizalt Can you check if you are able to do Kinit with the below commands?

klist -kt /etc/security/keytabs/nm.service.keytab 
kinit -kt /etc/security/keytabs/nm.service.keytab  <COPY principal for above output>

If the above command fails, regenerating the keytab for nodemanager should fix the issue.

[#3]

Thks @Scharan the repply
Yes, I can like below

root@slave1:~# klist -kt /etc/security/keytabs/nm.service.keytab
Keytab name: FILE:/etc/security/keytabs/nm.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 06/11/2024 11:05:54 nm/[email protected]
   2 06/11/2024 11:05:54 nm/[email protected]
root@slave1:~#

[#4]

Can you try regenerating the keytab,  seems to be the issue with kvno id

[#5]

I tried command kinit to make sure the password is correct, but message kinit is " password incorrect while getting initial credential" like below

root@master1:~# kinit nm/[email protected]
Password for nm/[email protected]:
kinit: Password incorrect while getting initial credentials

What should recreate principal/change the password ?
Please give me suggestion, I'm sure the password is correct