question Re: Unable to start namenode after enabling Kerberos in Support Questions

Unable to start namenode after enabling Kerberos

umesh_kumbhar — Thu, 28 Dec 2017 21:32:02 GMT

I have distributed setup of Hadoop cluster with 2 NN and 3 DN. I have enabled Kerberos on the cluster as per the steps mentioned in security document using Ambari wizard. On last step of Wizard, Ambari trying to start the services but Name node services are not getting started. In the namenode log file I can see below error:

2017-12-28 07:24:11,727 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(194)) - caught exception initializing http://ip-***-***-**-**.us-east-1.ec2.aws.net:8480/getJournal?jid=krbhdfs&segmentTxId=2979&storageInfo=-63%3A516351869%3A0%3ACID-3a8b7973-c162-4cb5-abbc-d90a2738580b
java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://ip-***-***-**-**.us-east-1.ec2.aws.net:8480/getJournal?jid=krbhdfs&segmentTxId=2979&storageInfo=-63%3A516351869%3A0%3ACID-3a8b7973-c162-4cb5-abbc-d90a2738580b&user.name=nn/ip-100-122-218-159.us-east-1.ec2.aws.net@KERBTEST.COM, status: 403, message: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)

The keytab file details are as below

#klist -kte nn.service.keytab 
Keytab name: FILE:nn.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 12/28/2017 07:02:18 nn/ip-***-***-***-**.us-east-1.ec2.aws.net@KERBTEST.COM (aes256-cts-hmac-sha1-96) 
   1 12/28/2017 07:02:18 nn/ip-***-***-***-**.us-east-1.ec2.aws.net@KERBTEST.COM (aes128-cts-hmac-sha1-96) 
   1 12/28/2017 07:02:18 nn/ip-***-***-***-**.us-east-1.ec2.aws.net@KERBTEST.COM (arcfour-hmac) 
   1 12/28/2017 07:02:18 nn/ip-***-***-***-**.us-east-1.ec2.aws.net@KERBTEST.COM (des3-cbc-sha1) 
   1 12/28/2017 07:02:18 nn/ip-***-***-***-**.us-east-1.ec2.aws.net@KERBTEST.COM (des-cbc-md5) 
# klist -kte spnego.service.keytab 
Keytab name: FILE:spnego.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 12/28/2017 07:02:17 HTTP/ip-***-***-***-**.us-east-1.ec2.aws.net@KERBTEST.COM (aes256-cts-hmac-sha1-96) 
   1 12/28/2017 07:02:17 HTTP/ip-***-***-***-**.us-east-1.ec2.aws.net@KERBTEST.COM (aes128-cts-hmac-sha1-96) 
   1 12/28/2017 07:02:17 HTTP/ip-***-***-***-**.us-east-1.ec2.aws.net@KERBTEST.COM (arcfour-hmac) 
   1 12/28/2017 07:02:17 HTTP/ip-***-***-***-**.us-east-1.ec2.aws.net@KERBTEST.COM (des3-cbc-sha1) 
   1 12/28/2017 07:02:17 HTTP/ip-***-***-***-**.us-east-1.ec2.aws.net@KERBTEST.COM (des-cbc-md5)

HDP version : HDP-2.5.0.55

Re: Unable to start namenode after enabling Kerberos

rlevas — Fri, 29 Dec 2017 00:19:08 GMT

One of a few issues may be in play. First, make sure that the unlimited key JCE policy is installed. Then make sure that the krb5.conf file or the KRB5CCNAME environment variable is not forcing the ticket cache to be stored in a KEYRING facility - the ticket cache needs to be stored in a file. Finally, ensure DNS and reverse DNS name resolution is configured properly.

Let me know if you need detailed explanations on any of those.

Re: Unable to start namenode after enabling Kerberos

umesh_kumbhar — Fri, 29 Dec 2017 17:25:22 GMT

@Robert Levas

Thanks for the reply.

JCE policy is installed correctly

 $ java test_jce
 2147483647

Ticket cache is stored in file.

[libdefaults]
renew_lifetime = 7d
forwardable = true
default_realm = KERBTEST.COM
ticket_lifetime = 24h
dns_lookup_realm = false
dns_lookup_kdc = false
default_ccache_name = /tmp/krb5cc_%{uid}

DNS entries are also fine. Looked at journal node logs. The above error is thrown by journal node.

Re: Unable to start namenode after enabling Kerberos

umesh_kumbhar — Fri, 29 Dec 2017 21:09:08 GMT

Issue got resolved. I copied the namenode keytab file on journal node and restarted the JournalNode. After this started namenode. Looks like JournalNode was not able to decrypt data from namenode. @Robert Levas is this correct resolution?

Re: Unable to start namenode after enabling Kerberos

rlevas — Mon, 01 Jan 2018 22:50:29 GMT

You should not have had to manually copy anything, so I am confused as to what the issue was.

Re: Unable to start namenode after enabling Kerberos

umesh_kumbhar — Tue, 02 Jan 2018 11:29:52 GMT

JournalNode throws below error when NameNode is trying to read the Journal during startup.

status: 403, message: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)

So I copied the NameNode keytab on journalNode and the error got resolved.

Re: Unable to start namenode after enabling Kerberos

umesh_kumbhar — Wed, 10 Jan 2018 03:06:35 GMT

@Robert Levas can you confirm if the solution is valid, as I am facing same issue on other cluster as well, and not sure of workaround.

Re: Unable to start namenode after enabling Kerberos

rlevas — Wed, 10 Jan 2018 03:24:21 GMT

Your solution of copying keytab files around is not valid. There must be some cause for the missing keytab file.

Re: Unable to start namenode after enabling Kerberos

umesh_kumbhar — Thu, 11 Jan 2018 17:54:25 GMT

Thanks @Robert Levas. I later figured out that "spnego.service.keytab" requires 444 access on all Journal node. Once I changed the mode to 444, an restarting the journalnode, NameNode started working.