all service stopped & failed to start after enabling kerberos

[#1]

kerb-namenode-logs.txt

Tried to enable kerberos manual way using HDP security guide.

After enabling kerberos while following steps , it stopped all services but failed to start any service and giving error.

Kindly find the attached file for reference.

P.S: in a kerberos installation step while giving the credentials of admin principal its working fine.

plz help if possible...

@kkulkarni

[#2]

@hardik desai

1. Can you quickly check if you have correct 755 permission on the keytab directory?

2. Are you manually able to do the kinit? (Just for testing)

# kinit  -kt /etc/security/keytabs/nn.service.keytab  nn/slave0.ns1.com@ns1.com
# klist

.

Also which version of ambari is it?

[#3]

@hardik desai

There is a authentication problem ! Login failure for nn/slave0.ns1.com@ns1.com from keytab /etc/security/keytabs/nn.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

klist -kt /etc/security/keytabs/nn.service.keytab

The above output should give you the valid principals to use see below extract

# klist -kt /etc/security/keytabs/nn.service.keytab
Keytab name: FILE:/etc/security/keytabs/nn.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 02/02/17 23:00:12 nn/kdc or admin_server@REALM
   1 02/02/17 23:00:12 nn/kdc or admin_server@REALM
   1 02/02/17 23:00:12 nn/kdc or admin_server@REALM
   1 02/02/17 23:00:12 nn/kdc or admin_server@REALM
   1 02/02/17 23:00:12 nn/kdc or admin_server@REALM 

Now from the above get a valid Kerberos ticket

# kinit -kt /etc/security/keytabs/nn.service.keytab nn/kdc or admin_server@REALM 

Validate that you got a ticket

# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: nn/kdc or admin_server@REALM
Valid starting     Expires            Service principal
03/07/17 16:45:00  03/08/17 16:45:00  krbtgt/REALM@REALM
        renew until 03/07/17 16:45:00

Get the correct values from /etc/krb5.conf

Now you can try restarting the nn !

[#4]

Hi @hardik desai - I've learned to take a broad view of messages about obtaining passwords for Kerberos. Here's a suggested list of things to check:

• Verify file permissions and ownership of the /etc/security/keytabs/*.keytab files is correct (mentioned above, see reference below). Keep in mind the local Linux user that runs a service must be able to access the appropriate keytab file.
• Verify you can manually kinit as the user principal (also mentioned above). I recommend performing an "su" to the local Linux account first (like "su hdfs" and then kinit to check the hdfs principal). Follow the steps mentioned above if you need help with kinit command.
• Use the klist -ket /etc/security/keytabs/<keytab-name> to verify your principal names are set correctly in the configuration. Ambari automated installation will handle all this for you, but a manual installation should be double checked.
• Verify your version of Java is correct for your HDP distribution. Supported versions are listed in the installation docs from docs.hortonworks.com under Getting Ready / Meet Minimum System Requirements / JDK requirements.
• Also note depending on your Java distribution you may have to install the Java Cryptography Extensions (JCE) that matched your JRE version. OpenJDK already includes what it needs, but Oracle JDK requires an additional installation.
• Older versions of HDP (like HDP 2.2 and I think the earlier HDP 2.3 versions) had some issues with certain Java / JCE combinations that caused problems with Kerberos. Again a double check of supported Java versions is recommended to be sure yours isn't too old or too new.

For reference, here's the permissions on the directory and some of the keytabs from a working installation:

[root@m1 ~]# ll /etc/security | grep keytabs
drwxr-xr-x. 2 root root 4096 Mar  6 11:34 keytabs

[root@m1 ~]# ll /etc/security/keytabs
-r--------. 1 hdfs hadoop  186 Mar  5 17:55 dn.service.keytab
-r--r-----. 1 hdfs hadoop  156 Mar  5 17:55 hdfs.headless.keytab
-r--r-----. 1 yarn  hadoop 190 Mar  5 17:55 hive.llap.zk.sm.keytab
-r--r-----. 1 hive hadoop  190 Mar  5 17:55 hive.service.keytab
-r--------. 1 mapred  hadoop  188 Mar  5 17:55 jhs.service.keytab
-r--------. 1 hdfs hadoop  186 Mar  5 21:51 jn.service.keytab
-r--------. 1 yarn hadoop  186 Mar  5 17:55 nm.service.keytab
-r--------. 1 hdfs hadoop  186 Mar  5 17:55 nn.service.keytab
-r--------. 1 yarn hadoop  186 Mar  5 17:55 rm.service.keytab
-r--r-----. 1 ambari-qa hadoop  166 Mar  5 17:55 smokeuser.headless.keytab
-r--r-----. 1 root hadoop  190 Mar  5 17:55 spnego.service.keytab
-r--------. 1 yarn hadoop  190 Mar  5 17:55 yarn.service.keytab
-r--------. 1 zookeeper hadoop  200 Mar  5 17:55 zk.service.keytab

Good luck!