Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issues with kerberizing a CDH 5.4 cluster (Enterprise edition but without Cloudera Navigator)

Highlighted

Issues with kerberizing a CDH 5.4 cluster (Enterprise edition but without Cloudera Navigator)

We have a CHD 5.4 cluster with namenode and resource manager HA with TLS configured.

 

I followed steps 1-8 and enabled Kerberos via SCM wizard (http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_intro_kerb.html )

( have cloudera's jdk and also installed JCE policy files)

 

After all the steps, even after kinit-ing with the HDFS super user principal, we are not able to perform any hdfs operations (map-red / hive etc. also do not work)

 

We noticed that the nodes do not have per-service keytab files which we would have expected SCM to generate.

(also journal node principals are not created)

 

The error we see is :

 

[root@cdh54-guru1 ~]# su - hdfs

 

[hdfs@cdh54-guru1 ~]$ kinit hdfs@ZALONILABS.COM

Password for hdfs@ZALONILABS.COM:

 

[hdfs@cdh54-guru1 ~]$ klist

Ticket cache: FILE:/tmp/krb5cc_493

Default principal: hdfs@ZALONILABS.COM

 

Valid starting     Expires            Service principal

08/05/15 15:39:26  08/06/15 15:39:26  krbtgt/ZALONILABS.COM@ZALONILABS.COM

renew until 08/12/15 15:39:26

 

 

[hdfs@cdh54-guru1 ~]$ hdfs dfs -ls /

 

15/08/05 15:39:37 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs@ZALONILABS.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed

15/08/05 15:39:38 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs@ZALONILABS.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed

15/08/05 15:39:38 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

15/08/05 15:39:40 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs@ZALONILABS.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed

15/08/05 15:39:44 WARN ipc.Client: Couldn't setup connection for hdfs@ZALONILABS.COM to cdh54-guru2.zalonilabs.com/10.11.12.202:8020

org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed

 

 

 

Is there a detailed config guide or any suggestion which may help ?

 

 

 

 

5 REPLIES 5
Highlighted

Re: Issues with kerberizing a CDH 5.4 cluster (Enterprise edition but without Cloudera Navigator)

New Contributor

Has there been a solution to this problem?

 

We have installed Kerberos on our system, and it seems to be a disaster.  There are so many hidden dependencies between the services.

Also the cloudera management console show all green, when in fact many services are failing.  

I tried to do a simple ls (sudo -u hdfs hdfs dfs -ls /user/cloudera/) at the command line and get the same error.

Highlighted

Re: Issues with kerberizing a CDH 5.4 cluster (Enterprise edition but without Cloudera Navigator)

Contributor

Some verification required :

 

1) Is your kerberos workstation configured with kdc properly?

2) Check the nameNode host and validate the connectivity from the client node

3) Refresh kerboeros credential  session , or try to use with keytab file.

 

I hope the above verification may hunt your exact issue.

 

 

Regards

-Khirod

Highlighted

Re: Issues with kerberizing a CDH 5.4 cluster (Enterprise edition but without Cloudera Navigator)

New Contributor

"1) Is your kerberos workstation configured with kdc properly?"

Exactly how do I test kdc ?

 

"2) Check the nameNode host and validate the connectivity from the client node"

Exactly how do I "validate the connectivity"?

 

"3) Refresh kerboeros credential  session , or try to use with keytab file."

Exactly how do I "refresh kerveros cerdential session"?

 

Re: Issues with kerberizing a CDH 5.4 cluster (Enterprise edition but without Cloudera Navigator)

New Contributor

1) Is your kerberos workstation configured with kdc properly?

If you have installed kerberos, type

kadmin

You will be taken to kadmin prompt.

Then you can check the principals using command

listprincs

You may also add your own principal say myusername@myrealm.

quit from kdmin

and check if you are able to get kerberos ticket, using

kinit username

if KDC is working fine, you will prompted for password and kerberos ticket will be granted.

You can check list of kerberos tickets using

klist -l

 

2) Check the nameNode host and validate the connectivity from the client node"

Once you get kerebros ticket, check contents of Hadoop DFS using

hadoop fs -ls

 

3) Refresh kerboeros credential  session , or try to use with keytab file."

In case, you need to refresh/regenerate kerberos credentials, using Cloudera Manager, go to

Administration-->Kerberos-->Credentials

Check credentials to be regenerated and click on Regenerate Selected.

Highlighted

Re: Issues with kerberizing a CDH 5.4 cluster (Enterprise edition but without Cloudera Navigator)

Explorer

Hi Guru,

 

I too faced the same issue and once after replacing the Java policy jar files with the Unlimited JCE jar files resolved the issue.

 

You can find these files under the java installation direcotry as below.

 

/usr/java/jdk1.7.0_25/jre/lib/security

 

Thanks,

Cibi

Don't have an account?
Coming from Hortonworks? Activate your account here