Fix was to remove or move the urika cache directory from all the nodes with (computes in my case).  Seems that these directories will get re-created during a run.

Bug when you go from simlpe AUTH to kerberos AUTH; the cache directories will not work if created under simple AUTH.