Kerberos: Changed Realm now hadoop KMS won't authenticate users.

[#1]

I'm sure something needs to be re-initialized but what needs to be re-inialized? Point me to docs and I"ll be happy.

[#2]

How have you changed the realm? It should be working if the krb.conf is correctly changed and on the kdc the kdc.conf. You might also need to change the kadmin config.

Maybe this helps: https://docs.oracle.com/cd/E19683-01/806-4078/6jd6cjrvc/index.html

[#3]

And i just forgot: you need to export all keytabs, as the realm name is included there

[#4]

You need to restart the KDC server and also the KMS server to get the latest configuarion changes.

[#5]

I rebooted the server several times. So I think it's fair to say I restarted it.

[#6]

@Matt Andruff

This is a bit tricky question

How did you change the REALM?

Did you disable Kerberos before changing the REALM? If so was it successful?

After responding to the above then I can have a better picture and probably help out.

[#7]

I used a different KDC server that was already setup.

I changed the configuration files that I know about to change the Realm. (KMS does not have a REALM parameter which makes me suspect it not using the correct REALM.)

I generated new keytabs for all components.

I pointed the krb5.conf to the new KDCS server.

[#8]

I might be misled, but in KMS if you enabled Kerberos the file kms-site.xml should contain entries like below. I think you will have a different location for the keytab. In the example the config does not mention the Kerberos realm, but still the keytab contains the realm (in the example below it is ${user.home}/kms.keytab). You need to export a new keytab for the principal configured (here http/localhost) and then copy it to ${user.home}/kms.keytab, afterwards KMS must be restartet.

As long as the default realm is configured the principal will be http/localhost@DEFAULTREALM. But if the keytab is exported for http/localhost@OLDREALM it will not find a valid key in the keytab.

You can check it with (path and principal as for the example) kinit http/localhost -k -t ${user.home}/kms.keytab on the KMS node.

   <property>
     <name>hadoop.kms.authentication.type</name>
     <value>kerberos</value>
   </property>

   <property>
     <name>hadoop.kms.authentication.kerberos.keytab</name>
     <value>${user.home}/kms.keytab</value>
   </property>

   <property>
     <name>hadoop.kms.authentication.kerberos.principal</name>
     <value>HTTP/localhost</value>
   </property>

   <property>
     <name>hadoop.kms.authentication.kerberos.name.rules</name>
     <value>DEFAULT</value>
   </property>

[#9]

To change the realm:

I used a different KDC server that was already setup.

I pointed the krb5.conf to the new KDCS server.

I changed the configuration files that I know about to change the Realm. (KMS does not have a REALM parameter which makes me suspect it not using the correct REALM.)

I generated new keytabs for all components. (@Geoffrey Shelton Okot @Harald Berghoff @Sankaru Thumuluru)