Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Exception Starting MapRed Job History Server after Kerberos

Highlighted

Exception Starting MapRed Job History Server after Kerberos

New Contributor

I am unable to start the Job History Server after kerberizing the cluster. I keep getting this exception everytime I try to start it.

at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1089)
	... 6 more
2017-11-21 12:43:06,984 FATAL hs.JobHistoryServer (JobHistoryServer.java:launchJobHistoryServer(224)) - Error starting JobHistoryServer
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: History Server Failed to login
	at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:128)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:221)
	at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:231)
Caused by: java.io.IOException: Login failure for jhs/HOST@KRB_HOST from keytab /etc/security/keytabs/jhs.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user


	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1098)
	at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:307)
	at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.doSecureLogin(JobHistoryServer.java:175)
	at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:126)
	... 3 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user


	at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1089)
	... 6 more

4 REPLIES 4

Re: Exception Starting MapRed Job History Server after Kerberos

Super Mentor

@Theyaa Matti

Please check if the "Job History Server" host is using the correct hostname in "lowercase" format for the following properties.

mapreduce.jobhistory.address
mapreduce.jobhistory.webapp.address

.

If above hostname is correct then please check if the keytab is correct? Means does it has correct hostname/principal name in it? In order to check that please check the output of the following command:

# klist -kte /etc/security/keytabs/jhs.service.keytab

.


If you want to do it specifically for "Job History Server" then If the keytab shows correct principal and if you are able to do "kinit" with it properly, then in that You might want to "regenerate" keytabs from ambari UI to see if it fixes your issue. ("Regenerating Keytabs" will require a downtime to restart all the services)

.

.

Re: Exception Starting MapRed Job History Server after Kerberos

New Contributor

Thank you @Jay Kumar SenSharma for the quick response. I also noticed that yarn resource managers are not able to communicate on the 8088 port due to having dual networks on the cluster. Kerberos seems to have added the internal network and did not add the external one, which yarn is trying to communicate with.

How would I add the external hostname/IP to kerberos also?

Re: Exception Starting MapRed Job History Server after Kerberos

Super Mentor

@Theyaa Matti

You can make the necessary hostname Vs IPAddress mapping inside your "/etc/hosts" file or all host in the cluster (or in case of DNS server please make those changes to the DNS mapping) and then restart all the services once from ambari server then from ambari UI perform "Regenerate Keytabs" operation so that the new keytabs can be generated with the correct hostname details.

Re: Exception Starting MapRed Job History Server after Kerberos

New Contributor

@Jay Kumar SenSharma

I have both hostnames in the /etc/hosts file, but when I go to the LDAP server I see principal for only one of them. I have dual homing on the cluster, the public name containt "-a" as a suffix to the shorname.

So for example the hostname could be host.test.com, the public one would be host-a.test.com. Both have different IP addresses.

Any idea how can I ask kerberos to create both principals please?