Member since
10-28-2016
392
Posts
7
Kudos Received
20
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2336 | 03-12-2018 02:28 AM | |
3622 | 12-18-2017 11:41 PM | |
2565 | 07-17-2017 07:01 PM | |
1760 | 07-13-2017 07:20 PM | |
5303 | 07-12-2017 08:31 PM |
01-25-2017
06:36 AM
@mqureshi - somehow i don't see the property - hbase.zookeeper.property.datadir in file hbase-site.xml attaching the file - hbase-site.xml (location - /etc/hbase/conf/hbase-site.xml) any ideas ? i'm on the HDP 2.4 sandbox.hbase-site.xml
... View more
01-25-2017
02:38 AM
Pls note - i'm able to manually create Hive table in encryption zone, and add data to the table. However, the 'create as ' command - is giving access error.
... View more
01-25-2017
02:37 AM
@Mahesh M. Pillai, @svenkat - looping you in, any ideas on what needs to be done on this ?
... View more
01-25-2017
02:36 AM
Hell, i'm evaluating & implementing creating Hive table & loading data when the Hive table is pointing to HDFS encryption zone. Here are the details -> - created hdfs location -> /encrypt/hive - created encryption zone - changed the scrachdir to location in encryption zone -> /encrypt/hive/tmp & provided permission 777 - given access to user - hive to hdfs location & key - created Hive table using following command -> create table testtable2 location '/encrypt/hive/testtable2' as select * from sample_07 limit 5; I get the error shown below, Any ideas ? Attaching the Ranger permissions screenshots. screen-shot-2017-01-24-at-30823-pm.png screen-shot-2017-01-24-at-62538-pm.png ---------------------------------------------- NFO : Moving data to: /encrypt/hive/testtable2 from
hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001
ERROR : Failed with exception Unable to move source
hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001
to destination /encrypt/hive/testtable2
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001
to destination /encrypt/hive/testtable2 at
org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2692) at
org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:106) at
org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:223) at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1720) at
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1477) at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113) at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.apache.hadoop.ipc.RemoteException(java.io.IOException):
/apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001
can't be moved into an encryption zone.
... View more
Labels:
01-25-2017
02:28 AM
@Mahesh M. Pillai, @svenkat - any ideas on this ?
... View more
01-25-2017
02:27 AM
Hi - i'm trying to evaluate & implement Data at Rest encryption for HBase. here is what is done -> - created folder /encrypt_hbase1/hbase - created Encryption zone using key - testkeyfromcli, path - /encrypt_hbase1 - added folders /encrypt_hbase1/hbase/staging, /encrypt_hbase1/hbase/data - made the following changes to properties in hbase-site,xml, to point Hbase to encrypted locations. hbase.rootdir => hdfs://sandbox.hortonworks.com:8020/encrypt_hbase1/hbase/data hbase.bulkload.staging.dir => /encrypt_hbase1/hbase/staging - added hbase to have access to locations under /encrypt_hbase1 (recursive)- using Ranger - Added hbase access to key - testkeyfromcli using Ranger I restarted Hbase using Ranger, and it starts up. However, when i try to access the tables (using command - list), the region server is shutting down, and it errors out. Any ideas on what needs to be done ? attached screen-shots of Ranger policies for HDFS location & key screen-shot-2017-01-24-at-62538-pm.png
screen-shot-2017-01-24-at-62459-pm.png ---------------------------------------------------------------- hbase(main):003:0> list
TABLE
ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2314)
at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:853)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:53136)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
List all tables in hbase. Optional regular expression parameter could
be used to filter the output. Examples:
hbase> list
hbase> list 'abc.*'
hbase> list 'ns:abc.*'
hbase> list 'ns:.*'
... View more
01-24-2017
11:33 PM
@Mahesh M. Pillai, @svenkat - getting one more issue when i create a table using command below -> Any ideas on how to fix this ? > create table testtable2 location '/encrypt/hive/testtable2' as select * from sample_07 limit 5; ---------------------------------------------- INFO : Moving data to: /encrypt/hive/testtable2 from hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001
ERROR : Failed with exception Unable to move source hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001 to destination /encrypt/hive/testtable2
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001 to destination /encrypt/hive/testtable2
at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2692)
at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:106)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:223)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1720)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1477)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): /apps/hive/warehouse/.hive-staging_hive_2017-01-24_23-28-52_250_4192737411546010800-4/-ext-10001 can't be moved into an encryption zone.
... View more
01-24-2017
11:17 PM
@Mahesh M. Pillai, @svenkat - was able to fix this, by adding permission to user - 'hive' to 'DECRYPT_EEK' on 'testkeyfromcli'
... View more
01-24-2017
11:11 PM
@Mahesh M. Pillai , @svenkat - seems i'm not able to get the user - hive to insert data into hive table, although i've provided access to user - 'hive' using Ranger Here are the details -> - created hdfs location -> /encrypt/hive - created encryption zone - changed the scrachdir to location in encryption zone -> /encrypt/hive/tmp & provided permission 777 - create table 'testhive' in location '/encrypt/hive' 0: jdbc:hive2://sandbox.hortonworks.com:10000> create table testhive (rno int, fname string, lname string) location '/encrypt/hive/testhive'; - when i try to add row to the table, it gives the following error -> ------------------------------------------ 0: jdbc:hive2://sandbox.hortonworks.com:10000> insert into testhive values(1, 'karan', 'alang');
Error: Error while compiling statement: FAILED: SemanticException [Error 10293]: Unable to create temp file for insert values User:hive not allowed to do 'DECRYPT_EEK' on 'testkeyfromcli' (state=42000,code=10293) ----------------------------------------------- Attached is the screenshot of Ranger permissions user - hive, location /encrypt/hive/ screen-shot-2017-01-24-at-30823-pm.png Pls note - value of hadoop.kms.blacklist.DECRYPT_EEK in -> /etc/ranger/kms/conf/dbks-site.xml <property>
<name>hadoop.kms.blacklist.DECRYPT_EEK</name>
<value>hdfs</value>
</property>
... View more
01-24-2017
02:45 AM
@Kuldeep Kulkarni - any ideas on this ?
... View more