Support Questions
Find answers, ask questions, and share your expertise

DEBUG hdfs.DFSClient: No KeyProvider found.

Rising Star

Hi,

We are using Ambari-2.0.2 and HDP-2.24.2

Our cluster is Kerberized using the AD Kerberos.

It was working fine until Java upgrade to JDK-1.7_80 on teh hadoop servers. There were no changes on the AD.

We started seeing below errors when try to invoke hdfs services.

DEBUG hdfs.DFSClient: No KeyProvider found.

15/12/17 13:22:14 DEBUG security.UserGroupInformation: PrivilegedActionException as:hdfs@HADOOP.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): Failure to initialize security context 15/12/17 13:22:14 DEBUG security.UserGroupInformation: PrivilegedAction as:hdfs@HADOOP.COM (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:644) 15/12/17 13:22:14 WARN ipc.Client: Couldn't setup connection for hdfs@HADOOP.COM to namenode1.hadoop.com/10.0.0.1:8020 15/12/17 13:22:14 DEBUG security.UserGroupInformation: PrivilegedActionException as:hdfs@HADOOP.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for hdfs@HADOOP.COM to namenode1.hadoop.com/10.145.4.30:8020 15/12/17 13:22:14 DEBUG ipc.Client: closing ipc connection to namenode1.hadoop.com/10.145.4.30:8020: Couldn't setup connection for hdfs@HADOOP.COM to namenode1.hadoop.com/10.145.4.30:8020 java.io.IOException: Couldn't setup connection for hdfs@HADOOP.COM to namenode1.hadoop.com/10.145.4.30:8020

I have copied JCE files in security folder after new Java installation.

I tried regenarting Kerberos princiapsl and Keytabs and restart servcies. They are up for a while and breaking with above errors again.

Please advice.

5 REPLIES 5

Rising Star

2015-12-17 14:02:08,491 WARN ipc.Server (Server.java:saslProcess(1291)) - Auth failed for 10.0.0.1:45502:null (Failure to initialize security context) 2015-12-17 14:02:08,492 INFO ipc.Server (Server.java:doRead(771)) - Socket Reader #1 for port 8020: readAndProcess from client 10.0.0.1 threw exception [javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)]]

What was the version of Java before you upgrade Java?

That's not an error I've got on my list of common errors; "failed to find tgt", which the generic "you aren't authorized" response.

Have a look at the debugging options and run with

export HADOOP_JAAS_DEBUG=true

and on the command, -Dsun.security.krb5.debug=true

That can help provide more details.

now, you say it works "for a while" and then stops. Does that mean it runs for 24h, 48h, 72h or some number like that?

Mentor

@Venkata Sridhar Gangavarapu are you still having issues with this? Follow advice from Steve to debug the issue. If anyone can help you fix it, it's him.

Mentor

@Venkata Sridhar Gangavarapu has this been resolved? Please accept best answer or provide your own solution.