Support Questions

Find answers, ask questions, and share your expertise

Unable to upload new files to encrypted zone in HDFS

avatar
Contributor

We have an encryption zone set up around a file structure for an application. We are trying to load the files into hdfs in this encrypted zone but we continue to get the following error:

GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

I have a valid Kerberos ticket and we opened up the KMS acls to allow all users to DECRYPT_EEK just to make sure we did not have an ACL setting wrong

 

Does anybody else have experience with this issue? 

1 ACCEPTED SOLUTION

avatar
Mentor
Thanks I'm certain you're hitting the same error as HADOOP-12559, given the
AuthenticationException is coming at write-time, and from the client
package that's used for HTTP work - indicating that the NN is unable to
contact the KMS.

You'll also likely observe this error only much after a NameNode restart
period (but that it works immediately after NN restart), and that it may go
away after one day or so, only to return again, which is inline with
HADOOP-12559's
behaviour within the NameNode.

The bug-fix update of 5.5.x or any minor upgrade to the newer releases
should solve this up.

View solution in original post

10 REPLIES 10

avatar
Explorer

FWIW I seem to have found a solution.

 

I had added  a call to 

ugi.checkTGTAndReloginFromKeytab()

but it hadn't worked.

 

Later in debugging I found that that call was trying to renew the Proxy User, not the underlying principal.

 

I changed the call so that it would get the principal's ugi and call the same method on that and now it seems to work.

 

There are still outstanding questions, though, if anyone cares to investigate further:

 

  • Why was this only necessary for D.A.R.E. ?  All other ops (hdfs, Hive, yarn, etc.) continued working and renewing krbtgt's perpetually
  • Was the upgrade of CDH needed or would it have continued working with the older version?