HDFS Encryption at Rest - creating hive table gives error

karan_alang1 — Wed, 25 Jan 2017 10:36:42 GMT

Hell, i'm evaluating & implementing creating Hive table & loading data when the Hive table is pointing to HDFS encryption zone.

Here are the details ->

- created hdfs location -> /encrypt/hive

- created encryption zone

- changed the scrachdir to location in encryption zone -> /encrypt/hive/tmp & provided permission 777

- given access to user - hive to hdfs location & key

- created Hive table using following command ->

create table testtable2 location '/encrypt/hive/testtable2' as select * from sample_07 limit 5;

I get the error shown below, Any ideas ?

Attaching the Ranger permissions screenshots.

screen-shot-2017-01-24-at-30823-pm.png

screen-shot-2017-01-24-at-62538-pm.png

----------------------------------------------

NFO : Moving data to: /encrypt/hive/testtable2 from hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001 ERROR : Failed with exception Unable to move source hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001 to destination /encrypt/hive/testtable2 org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001 to destination /encrypt/hive/testtable2 at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2692) at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:106) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:223) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1720) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1477) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154) at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71) at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): /apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001 can't be moved into an encryption zone.

Re: HDFS Encryption at Rest - creating hive table gives error

karan_alang1 — Wed, 25 Jan 2017 10:37:34 GMT

@Mahesh M. Pillai, @svenkat - looping you in, any ideas on what needs to be done on this ?

Re: HDFS Encryption at Rest - creating hive table gives error

karan_alang1 — Wed, 25 Jan 2017 10:38:40 GMT

Pls note - i'm able to manually create Hive table in encryption zone, and add data to the table.

However, the 'create as ' command - is giving access error.

Re: HDFS Encryption at Rest - creating hive table gives error

karan_alang1 — Thu, 26 Jan 2017 07:05:26 GMT

hive.exec.stagingdir was already set to - /encrypt/hive/tmp/

also, scrachdir to location in encryption zone -> /encrypt/hive/tmp & provided permission 777

There is an additional variable that was to be changed ->

hive.metastore.warehouse.dir - I changed this from existing value (/apps/hive/warehouse) to

location in the encrypted zone -> /encrypt/hive, and this problem is fixed.

----------------------------------------------------------------------------------------------------------

INFO : Moving data to: /encrypt/hive/testtable2 from hdfs://sandbox.hortonworks.com:8020/encrypt/hive/.hive-staging_hive_2017-01-25_22-54-41_396_5265658181234256688-1/-ext-10001 INFO : Table default.testtable2 stats: [numFiles=1, numRows=5, totalSize=211, rawDataSize=206] No rows affected (47.001 seconds)

question Re: HDFS Encryption at Rest - creating hive table gives error in Support Questions

HDFS Encryption at Rest - creating hive table gives error

Re: HDFS Encryption at Rest - creating hive table gives error

Re: HDFS Encryption at Rest - creating hive table gives error

Re: HDFS Encryption at Rest - creating hive table gives error