ISSUE: While performing unkerberizing cluster all services were down and nothing was coming up. Also the unkeberized cluster step failed. The start of services was failed.
Tried to manually start Namenodes which came up but the status was not displayed correctly in Ambari UI. The journal node were not able to start and was failing with error as shown below.
ERROR: Screenshot is attached below
Journal node error:
ROOT CAUSE:
There were multiple issue as below -
1. From the JN error it says "missing spnego keytab". From the error It seems the kerberos was not properly disabled on cluster.
2. As checked in hdfs-site.xml the property "hadoop.http.authentication.type" was set to kerberos.
3. Oozie was not able to detect active namenode, since the property "hadoop.http.authentication.simple.anonymous.allowed" was set to false.
RESOLUTION:
1. Setting hadoop.http.authentication.type to simple in hdfs-site.xml, HDFS was able to restart
2. Setting the property hadoop.http.authentication.simple.anonymous.allowed=true in hdfs-site.xml oozie was able to detect active namenode and also namenode status was corrrectly displayed in namenode UI.
