Cluster Wipe/Rebuild/Kerberization

[#1]

We rebuilt a kerberized cluster with new hostnames. Should we give it a new cluster name or can we use the same cluster name that is already in active directory. What's the cleanest way to do this without problems in Active Directory? Cluster will be in same hadoop OU.

[#2]

@Debra

Have you recreated the same REALM & password? remember when you run the netdom trust you specified the HADOOP and AD realms with a password

And also while creating the "Trust User" with the wiped cluster you created a principal in the KDC that you recreate linking the KDC to the AD.

There is also the keytab KVNO associated with the wiped out cluster could cause a nightmare, So it would be reasonable to create a new cluster name and entries in the AD and later delete the old entries.

[#3]

Ambari will happily create or reset accounts in the Active Directory. So if there is an existing account, say for hdfs-myclustername@REALM, Ambari will find the relevant account, change the password, and create the keytab file for it. This is good and bad. Good, because it makes your scenario easy to deal with so you don't have to go and clean out the Active Directory. Bad, because don't have to go and clean out the Active Directory and you can accidentally change the password out from under an existing keytab file that may still be in use. So care must be used.

One thing to keep in mind is that the principal name (userPrincipalName) must be unique. Active Directory 2012 will enforce this; however, I believe that Active Directory 2008 does not. Being that you plan to use the same container (OU), this will help. However if you used a different container, you could see collision issues.

That said, you are welcome to change the cluster name or keep it the same. In any case you will have orphaned accounts since the hostnames will be different and the service principals contain a hostname. This is not an issue for Ambari or the hadoop cluster, however your security policies may not allow that.

Finally, if you disabled Kerberos before killing the old cluster, Ambari should clean up the principals it manages. In Ambari 3.0.0, this should be more granular and the relevant managed principals will be clean up when services, components, and hosts are removed.

[#4]

Hi Robert,

Thanks for the quick and detailed response. In our initial planning, we didn't know that the host names would change and didn't disable Kerberos before wiping the cluster. So what issues will this cause?

Thanks,

[#5]

There should not be any issues related to Ambari or the Hadoop cluster. I think the only effect you will see is that there are orphaned accounts in the Active Directory.

[#6]

Got it. For some reason ambari created a kerberos service check keytab with a date in the name. I renamed this on all hosts and started from scratch with kerberization and it worked this time using same cluster name.

Thanks,