Unable to put files in HDFS Encrypted Zone

[#1]

SYMPTOMS: Although valid Kerberos ticket is available, we are unable to put files in HDFS encrypted zone. If we restart/failover namenode, then we are able to put files with the same ticket/credentials without having to get a new ticket. Below is the demo of the issue where /tmp/user1 is the encrypted zone and user has permission to that zone:

[root@test ~]# su - user Last login: Thu Oct 13 13:03:24 EDT 2016 on pts/57 -bash-4.2$ id uid=11516(user) gid=5000(bns) groups=5000(bns),1520(cmtsuser),1800(admin),4534(edgegrp),4535(edgedgrp),4536(k2tstgrp),8242(ftallocctxd),8243(ftallocctxu),15113(hdpadm) 

-bash-4.2$ kinit Password for user@123.EXAMPLE.COM: 

-bash-4.2$ klist 
Ticket cache: FILE:/tmp/krb5cc_11516 
Default principal: user@123.EXAMPLE.COM
Valid starting Expires Service principal 
10/14/2016 07:23:51 10/14/2016 17:23:51 krbtgt/123.EXAMPLE.COM@EXAMPLE.COM 
renew until 10/21/2016 07:23:48 
-bash-4.2$ hadoop fs -put file1 /tmp/user1/file_1 
put: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) 
-bash-4.2$ 
-bash-4.2$ hadoop fs -put file1 /tmp/file_1 
-bash-4.2$ hadoop fs -cat /tmp/file_1 
diana 
-bash-4.2$

ROOT CAUSE: Service Delegation Token (DT) renewal was not working because the customer code misses the token renewer class in KMS. After enabling Hadoop KMS, the cluster can only work normally until the configured time in property hadoop.kms.authentication.delegation-token.renew-interval.sec is reached. The config does not exists in customer code, and the default one is 86400 essentially 1 day.

SOLUTION: Following options are provided:

1. If the customer plans to upgrade to newer version (e.g. HDP 2.5), the problem does not exist as all fixes will be included.
2. Otherwise a hotfix can be provided for them to include those fixes. Please raise a support case for the same.

REFERENCE:

https://issues.apache.org/jira/browse/HADOOP-13155