Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)

SYMPTOMS: Although valid Kerberos ticket is available, we are unable to put files in HDFS encrypted zone. If we restart/failover namenode, then we are able to put files with the same ticket/credentials without having to get a new ticket. Below is the demo of the issue where /tmp/daniel is the encrypted zone and user has permission to that zone:

[root@test ~]# su - djianu Last login: Thu Oct 13 13:03:24 EDT 2016 on pts/57 -bash-4.2$ id uid=11516(djianu) gid=5000(bns) groups=5000(bns),1520(cmtsuser),1800(admin),4534(edgegrp),4535(edgedgrp),4536(k2tstgrp),8242(ftallocctxd),8243(ftallocctxu),15113(hdpadm) 

-bash-4.2$ kinit Password for djianu@123.EXAMPLE.COM: 

-bash-4.2$ klist 
Ticket cache: FILE:/tmp/krb5cc_11516 
Default principal: djianu@123.EXAMPLE.COM
Valid starting Expires Service principal 
10/14/2016 07:23:51 10/14/2016 17:23:51 krbtgt/123.EXAMPLE.COM@EXAMPLE.COM 
renew until 10/21/2016 07:23:48 
-bash-4.2$ hadoop fs -put file1 /tmp/daniel/file_1 
put: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) 
-bash-4.2$ 
-bash-4.2$ hadoop fs -put file1 /tmp/file_1 
-bash-4.2$ hadoop fs -cat /tmp/file_1 
diana 
-bash-4.2$

ROOT CAUSE: Service Delegation Token (DT) renewal was not working because the customer code misses the token renewer class in KMS. After enabling Hadoop KMS, the cluster can only work normally until the configured time in property hadoop.kms.authentication.delegation-token.renew-interval.sec is reached. The config does not exists in customer code, and the default one is 86400 essentially 1 day.

SOLUTION: Following options are provided:

  1. If the customer plans to upgrade to newer version (e.g. HDP 2.5), the problem does not exist as all fixes will be included.
  2. Otherwise a hotfix can be provided for them to include those fixes. Please raise a support case for the same.

REFERENCE:

https://issues.apache.org/jira/browse/HADOOP-13155

https://hortonworks.jira.com/browse/BUG-68102

722 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎12-24-2016 10:21 PM
Updated by:
 
Contributors
Top Kudoed Authors