Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Not able tro access HDFS after enabling kerberos

Solved Go to solution

Not able tro access HDFS after enabling kerberos

Hi,

 

We are getting the below error whenever we are trying to access HDFS from console. This error is coming after enabling the kerberos authentication in the cluster. We are getting this error when we are trying to access from a customized user 'edtuser'. But we are not getting error when we are accessing error from 'root' user. and 'hdfs' user.Please find the error below:-

 

[edtuser@nladfmrvu11 ~]$
[edtuser@nladfmrvu11 ~]$ hadoop fs -ls /
19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:48:06 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 1 failover attempts. Trying to failover after sleeping for 610ms.
19/08/22 09:48:06 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:48:06 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 2 failover attempts. Trying to failover after sleeping for 1680ms.
19/08/22 09:48:08 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:48:08 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 3 failover attempts. Trying to failover after sleeping for 5206ms.
19/08/22 09:48:13 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:48:13 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 4 failover attempts. Trying to failover after sleeping for 10481ms.
19/08/22 09:48:24 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:48:24 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 5 failover attempts. Trying to failover after sleeping for 8347ms.
19/08/22 09:48:32 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:48:32 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 6 failover attempts. Trying to failover after sleeping for 13318ms.
19/08/22 09:48:45 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:48:45 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 7 failover attempts. Trying to failover after sleeping for 14341ms.
19/08/22 09:49:00 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:49:00 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 8 failover attempts. Trying to failover after sleeping for 17707ms.
19/08/22 09:49:17 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:49:17 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 9 failover attempts. Trying to failover after sleeping for 12086ms.
19/08/22 09:49:29 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:49:29 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 10 failover attempts. Trying to failover after sleeping for 11451ms.
19/08/22 09:49:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:49:41 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 11 failover attempts. Trying to failover after sleeping for 22482ms.
19/08/22 09:50:03 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:50:03 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 12 failover attempts. Trying to failover after sleeping for 22157ms.
19/08/22 09:50:25 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:50:25 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu12.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu12.mtn.co.za/10.244.8.53:8020 after 13 failover attempts. Trying to failover after sleeping for 20978ms.
19/08/22 09:50:46 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/22 09:50:46 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort nladfmrvu11.mtn.co.za:8020 , LocalHost:localPort nladfmrvu11.mtn.co.za/10.244.8.52:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over nladfmrvu11.mtn.co.za/10.244.8.52:8020 after 14 failover attempts. Trying to failover after sleeping for 16712ms.
^Z
[1]+ Stopped hadoop fs -ls /
[edtuser@nladfmrvu11 ~]$

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Not able tro access HDFS after enabling kerberos

Explorer

also check if the user can read from the keytab file

5 REPLIES 5

Re: Not able tro access HDFS after enabling kerberos

Cloudera Employee

Hi @pritam_konar ,

 

Verify whether you have valid TGT to access the HDFS or not.

To verify the same execute below command:

[edtuser@nladfmrvu11 ~]$ klist

 

If you find no TGT under /tmp/cache or the TGT is expired then get new TGT by executing below command below

 [edtuser@nladfmrvu11 ~]$ kinit

 

 

Highlighted

Re: Not able tro access HDFS after enabling kerberos

Please find the information about edtuser:-

 

[edtuser@nladfmrvu11 ~]$
[edtuser@nladfmrvu11 ~]$ klist
klist: No credentials cache found (filename: /tmp/krb5cc_5009)
[edtuser@nladfmrvu11 ~]$
[edtuser@nladfmrvu11 ~]$ ll /tmp/cache
ls: cannot access /tmp/cache: No such file or directory
[edtuser@nladfmrvu11 ~]$

Re: Not able tro access HDFS after enabling kerberos

Cloudera Employee

Are you able to do 

$ kinit

Re: Not able tro access HDFS after enabling kerberos

Explorer

can you 

#cat /tmp/krb5cc_5009

It looks like the user is taking some other users tickets. 

so run 

kdestroy

then kinit as the user

Re: Not able tro access HDFS after enabling kerberos

Explorer

also check if the user can read from the keytab file

Don't have an account?
Coming from Hortonworks? Activate your account here