question Re: Server has invalid Kerberos principal in Archives of Support Questions (Read Only)

Server has invalid Kerberos principal

openbusinessintelligence — Tue, 10 May 2016 16:22:59 GMT

After kerberising the cluster (HDP 2.3.2 on SLES 11.4) many services doesn't start anymore:

Nimbus, Storm Supervisor, HBase Master, Phoenix and the YARN node managers on the nodes other than the Resource Manager

In detail, the YARN log contain the following:

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Server has invalid Kerberos principal: rm/<res_mgr_host>@hdp23cluster; Host Details : local host is: "<nod_mgr_host>/<nod_mgr_ip>"; destination host is: "<res_mgr_host>":8025;

On <res_mgr_host> runs the YARN RM, on <nod_mgr_host> run an additional node manager

I've installed a standard kerberos server using zypper and succesfully configured kerberos in Ambari leaving all default values.

On all nodes is configured a proxy, additionally, the no_proxy system variable contains the list of hosts for which the proxy should be ignored: all node hosts + other hosts.

What could be wrong?

Re: Server has invalid Kerberos principal

sshimpi — Mon, 19 Aug 2019 08:23:41 GMT

Hi @Nicola Marangoni

Can you login to ambari and click on Kerberos->Advance and check if the principals are present /created for respective services properly ? Check the screenshot below -

Re: Server has invalid Kerberos principal

sshimpi — Tue, 10 May 2016 16:32:39 GMT

Were there any issue while kerberizing the cluster? Did it fininshed without any error ?

Re: Server has invalid Kerberos principal

rlevas — Tue, 10 May 2016 20:00:53 GMT

@Nicola Marangoni

Unless you changed this when editing your log entry for the post, your realm is incorrect. You have "hdp23cluster" as your realm when it should be in all upper case characters - "HDP23CLUSTER".

To change this, your best bet is to disable Kerberos and then re-enable Kerberos with the correct realm.

Re: Server has invalid Kerberos principal

openbusinessintelligence — Tue, 10 May 2016 20:43:29 GMT

@Sagar Shimpi: the user exists.

Re: Server has invalid Kerberos principal

openbusinessintelligence — Tue, 10 May 2016 20:45:54 GMT

@Robert Levas: The realm ist in lower case also in kerberos. Should I indeed enter it in uppercase in Ambari?

Re: Server has invalid Kerberos principal

rlevas — Tue, 10 May 2016 21:01:05 GMT

Technically, if the realm matches in the KDC, the /etc/krb5.conf file, and Ambari, all should work. But I have seen that the MIT Kerberos libraries tend to assume the realm is all uppercase - or maybe it is the internal Hadoop Kerberos logic.

You can check the MIT libray case by attempting to manually kinit and see if it works.

kinit -kt /etc/security/keytabs/rm.service.keytab  rm/<res_mgr_host>@hdp23cluster

In any case, I would disable Kerberos in Ambari, rebuild the KDC using the uppercase form of the realm, and then re-enable Kerberos. If it doesn't work after this, we can at least rule out the case-sensitivity issue.

Re: Server has invalid Kerberos principal

openbusinessintelligence — Tue, 10 May 2016 23:09:12 GMT

@Robert Levas

Thank you for the tip, after recreating and re-configuring everything with an uppercase realm, it still doesn't work.

However, I have noticed that the keytab rm.service.keytab is present on the RM host, but not in the other hosts.

Should the keytab be present on every host? If yes, than the automatic deployment of the keytabs doesn't work well.

Keytabs on the non-RM node:

dn.service.keytab
hbase.headless.keytab
hdfs.headless.keytab
knox.service.keytab
nfs.service.keytab
nm.service.keytab
nn.service.keytab
smokeuser.headless.keytab
spark.headless.keytab
spnego.service.keytab
zk.service.keytab

Keytabs on the RM node:

dn.service.keytab
hbase.headless.keytab
hdfs.headless.keytab
hive.service.keytab
jhs.service.keytab
nfs.service.keytab
nm.service.keytab
nn.service.keytab
oozie.service.keytab
rm.service.keytab
sbetp.headless.keytab
smokeuser.headless.keytab
spark.headless.keytab
spnego.service.keytab
yarn.service.keytab
zk.service.keytab

Re: Server has invalid Kerberos principal

openbusinessintelligence — Tue, 10 May 2016 23:53:30 GMT

@Robert Levas

The command

kinit -kt /etc/security/keytabs/rm.service.keytab  rm/<res_mgr_host>@hdp23cluster

works only on the RM-node, maybe because of the missing keytab.

After copying the rm.service.keytab on all nodes the command works in the console, but the node manager fails again with the same error "Server has invalid Kerberos principal".

Re: Server has invalid Kerberos principal

rlevas — Tue, 10 May 2016 23:54:25 GMT

@Nicola Marangoni

The keytabs are only distributed to the hosts on which they are needed. So I do not expect all keytab file to be distributed to all hosts.

Re: Server has invalid Kerberos principal

rlevas — Tue, 10 May 2016 23:59:51 GMT

I only expect a keytab file to work on the particular host it was distributed to. This is because the service principals have the hostname where the service is running embedded in its name. So it is not recommended to copy them around.

That said, you might want to make sure that the hostname of the hosts is being represented the same via the different mechanisms for getting the host's name.

For example, hostname -f should be the fully qualified domain name (FQDN) of the host and return the same FQDN that was used to register with Ambari.

Re: Server has invalid Kerberos principal

PranayV — Wed, 11 May 2016 03:40:01 GMT

This appears to be FQDN issue. Does your DNS resolution happen through a DNS server or hosts file? if it is hosts file make sure all nodes have fqdn followed by their assigned IP address.

Re: Server has invalid Kerberos principal

openbusinessintelligence — Wed, 11 May 2016 14:31:43 GMT

@Robert Levas @Pranay Vyas

Name resolution works over a DNS server, but Kerberos seems to ignore it.

Adding IP/Hosts to the /etc/hosts file seems to help, so thank you for the tip!

However, this doesn't solve the problem but generate a different error message:

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.authorize.AuthorizationException: User nm/msas6502i.msg.de@HDP23CLUSTER (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.yarn.server.api.ResourceTrackerPB, expected client Kerberos principal is nm/10.100.233.13@HDP23CLUSTER

Re: Server has invalid Kerberos principal

openbusinessintelligence — Wed, 11 May 2016 16:24:58 GMT

@Robert Levas @Pranay Vyas It was definitively a DNS problem: Kerberos can't use the DNS, it can resolve names only over /etc/hosts

Many thanks!

Re: Server has invalid Kerberos principal

openbusinessintelligence — Wed, 11 May 2016 16:45:12 GMT

I had to unkerberize and rekerberize the cluster, now it works!