Can't drop hive tables after moving hive into encryption zone.

vjiang — Wed, 07 Oct 2015 07:38:18 GMT

We are validating Hadoop TDE and have moved /apps and /data into encryption zones. The hive.metastore.warehouse.dir is set to /apps/hive/warehouse

There are 2 issues.

1. We cant delete hive tables because Trash is not in the encryption zone.

hive> CREATE DATABASE alan_test; 
OK 
Time taken: 3.027 seconds 
hive> use alan_test; 
OK 
Time taken: 0.315 seconds 
hive> CREATE TABLE a(id int); 
OK 
hive> DROP DATABASE alan_test; 
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. InvalidOperationException(message:Database alan_test is not empty. One or more tables exist.)

2. We can only delete files via hdfs dfs -rm .... if we use the -skipTrash option.

[root@devsl1 ~]# sudo -u ssn hdfs dfs -copyFromLocal TEST /apps/hive/ 
[root@devsl1 ~]# sudo -u ssn hdfs dfs -rm /apps/hive/TEST 
15/10/06 15:51:51 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes. 
rm: Failed to move to trash: hdfs://devsl1.eng.ssnsgs.net:8020/apps/hive/TEST: /apps/hive/TEST can't be moved from an encryption zone. 


[root@devsl1 ~]# sudo -u ssn hdfs dfs -rm -skipTrash /apps/hive/TEST 
Deleted /apps/hive/TEST

I think this is because Hive does not support HDFS TDE yet (HIVE-8065).

But just want to make sure it's the case and see if there is any workaround.

Re: Can't drop hive tables after moving hive into encryption zone.

deepesh1 — Wed, 07 Oct 2015 07:41:58 GMT

You can delete Hive tables by calling "drop table <tablename> purge;", this will skip the trash. If this is for testing purposes you can temporarily set fs.trash.interval to 0 and restart namenode. This will globally disable trash collection on HDFS so should only be employed during testing.

On your last question about the support of TDE feature, it was available starting HDP 2.3.

question Re: Can't drop hive tables after moving hive into encryption zone. in Archives of Support Questions (Read Only)

Can't drop hive tables after moving hive into encryption zone.

Re: Can't drop hive tables after moving hive into encryption zone.