Support Questions
Find answers, ask questions, and share your expertise

Unable to access Hadoop CLI after enabling Kerberos

Solved Go to solution
Highlighted

Unable to access Hadoop CLI after enabling Kerberos

Explorer

Hi all,

 

I've followed the following tutorial CDH Hadoop Kerberos, NameNode and DataNode are able to start properly and I'm able to see all the DataNode listed on the WebUI (0.0.0.0:50070). But I'm unable to access the Hadoop CLI. I've followed this tutorial Certain Java versions cannot read credentials cache, still I'm unable to use the Hadoop CLI.

 

[root@local9 hduser]# hadoop fs -ls /
20/11/03 12:24:32 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
20/11/03 12:24:32 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
20/11/03 12:24:32 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "local9/192.168.2.9"; destination host is: "local9":8020;
[root@local9 hduser]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_hVEAjWz
Default principal: hdfs/local9@FBSPL.COM

Valid starting       Expires              Service principal
11/03/2020 12:22:42  11/04/2020 12:22:42  krbtgt/FBSPL.COM@FBSPL.COM
        renew until 11/10/2020 12:22:12
[root@local9 hduser]# kinit -R
[root@local9 hduser]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_hVEAjWz
Default principal: hdfs/local9@FBSPL.COM

Valid starting       Expires              Service principal
11/03/2020 12:24:50  11/04/2020 12:24:50  krbtgt/FBSPL.COM@FBSPL.COM
        renew until 11/10/2020 12:22:12
[root@local9 hduser]# hadoop fs -ls /
20/11/03 12:25:04 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
20/11/03 12:25:04 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
20/11/03 12:25:04 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "local9/192.168.2.9"; destination host is: "local9":8020;

 

Any Help would be greatly appreciated. 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Unable to access Hadoop CLI after enabling Kerberos

Contributor

Hello @sace17 

 

It seems your problem is related to credential cache.

Per "https://bugzilla.redhat.com/show_bug.cgi?id=1029110", 

If the keyring ccache is changed from UID to username like below, it is not possible to get ticket as non-root user.
 default_ccache_name = KEYRING:persistent:%{username}

We have a KB article talks about the problem - https://community.cloudera.com/t5/board/article/ta-p/74262

 

Per KB article, CDH/Hadoop components do not fully support the advanced Linux feature KEYRING to store Keberos credentials.
Remove any global profile setting for environment variable KRB5CCNAME. If no type prefix is present, the FILE type is assumed, which is supported by CDH/Hadoop components.

 

Please remove/comment the section in /etc/krb5.conf file of all cluster nodes and that should solve your problem.  

 

Ref community post on the same problem here - https://community.cloudera.com/t5/Support-Questions/Kerberos-Cache-in-IPA-RedHat-IDM-KEYRING-SOLVED/...

 

Additional Reference: 

https://web.mit.edu/kerberos/krb5-1.12/doc/basic/ccache_def.html

 

Thank you 

View solution in original post

8 REPLIES 8
Highlighted

Re: Unable to access Hadoop CLI after enabling Kerberos

Cloudera Employee

Hi, Please check the below cloudera article it seems to be the same issue. 

https://community.cloudera.com/t5/Internal/Kerberos-Authentication-error-Fail-to-create-credential-6...

Highlighted

Re: Unable to access Hadoop CLI after enabling Kerberos

Explorer

@ChethanYM 

I'm unable to access the link that you shared.

I'm Getting a access denied to link, then it's opening a 404 page

Highlighted

Re: Unable to access Hadoop CLI after enabling Kerberos

Cloudera Employee

This error occurs when you have AES256 encryption enabled and you recently upgraded Java. Upgrading Java will overwrite the JCE policy files which include support for AES256 encryption. can you simply re-install your JCE policy jars and give a try?

Highlighted

Re: Unable to access Hadoop CLI after enabling Kerberos

Explorer

I didn't upgrade the java. Anyways I reinstalled the JCE jar but the issue remained the same. No luck.

Re: Unable to access Hadoop CLI after enabling Kerberos

Cloudera Employee

Can you remove the following line 'default_ccache_name = KEYRING:persistent:%{uid}' from the krb5.conf and run the hdfs dfs command?

Highlighted

Re: Unable to access Hadoop CLI after enabling Kerberos

Contributor
Highlighted

Re: Unable to access Hadoop CLI after enabling Kerberos

Explorer

Thanks, I'm able to access the Hadoop CLI after commenting out the line.

Highlighted

Re: Unable to access Hadoop CLI after enabling Kerberos

Contributor

Hello @sace17 

 

It seems your problem is related to credential cache.

Per "https://bugzilla.redhat.com/show_bug.cgi?id=1029110", 

If the keyring ccache is changed from UID to username like below, it is not possible to get ticket as non-root user.
 default_ccache_name = KEYRING:persistent:%{username}

We have a KB article talks about the problem - https://community.cloudera.com/t5/board/article/ta-p/74262

 

Per KB article, CDH/Hadoop components do not fully support the advanced Linux feature KEYRING to store Keberos credentials.
Remove any global profile setting for environment variable KRB5CCNAME. If no type prefix is present, the FILE type is assumed, which is supported by CDH/Hadoop components.

 

Please remove/comment the section in /etc/krb5.conf file of all cluster nodes and that should solve your problem.  

 

Ref community post on the same problem here - https://community.cloudera.com/t5/Support-Questions/Kerberos-Cache-in-IPA-RedHat-IDM-KEYRING-SOLVED/...

 

Additional Reference: 

https://web.mit.edu/kerberos/krb5-1.12/doc/basic/ccache_def.html

 

Thank you 

View solution in original post