Cloudera 5 Service Manager failed to start during Kerberos implementation

[#1]

Hi All,

Good news was that I was able to get CHD5.1.2 3-node cluster up and running using Cloudera Manager Installer; thanks for the help there. 

Now for my next adventure; I've created all the necessary Kerberos pricipals, keytab/krb5.conf files and scp'd them over to the nodes on the cluster... and then started the Kerberos set-up using Cloudera Manager.  Everything seemed to go very well... until after all configuration changes were made and the cluster was restated.  That's when disaster struck.  :-0

it seemed to choke on the restart of Cloudera Management Services starting with Activity manager onwards as seen below. 

So to provide more, here's a screenshot of the failed commands;

So now what I have is a cluster that will not start and there doesn't appear to be a clear way of rolling back the kerberos set up. 

I have 2 sets of questions;

• First, what keytab file does CM think is missing? If I create and add this keytab, how do I restart the process?

• Second, how do I recover from here? Do I have to restore a back-up and start all over again or is there a way for Cloudera Manager to roll back the Kerberos configurations?

Thank you in advance for your collective wisdom on this.

mit Freundlichen Grüßen (with Friendly Greetings),
     Jan
Jan Peters

[#2 · Accepted solution]

Okay,

I'm today's cheap entertainment...  problem solved! 😄

Someone disabled the krb5kdc service on our KDC server (AAARRRGGGHHH)  😮

so a simple service krb5kdc start there allowed the rest of the Kerberos set up to complete and we're back up using Kerberos. 

mit Freundlichen Grüßen (with Friendly Greetings),
     Jan
Jan Peters