HDFS Encryption Zone - HBase shutting down

[#1]

Hi - i'm trying to evaluate & implement Data at Rest encryption for HBase.

here is what is done ->

- created folder /encrypt_hbase1/hbase

- created Encryption zone using key - testkeyfromcli, path - /encrypt_hbase1

- added folders /encrypt_hbase1/hbase/staging, /encrypt_hbase1/hbase/data

- made the following changes to properties in hbase-site,xml, to point Hbase to encrypted locations.

hbase.rootdir => hdfs://sandbox.hortonworks.com:8020/encrypt_hbase1/hbase/data

hbase.bulkload.staging.dir => /encrypt_hbase1/hbase/staging

- added hbase to have access to locations under /encrypt_hbase1 (recursive)- using Ranger

- Added hbase access to key - testkeyfromcli using Ranger

I restarted Hbase using Ranger, and it starts up.

However, when i try to access the tables (using command - list), the region server is shutting down, and it errors out.

Any ideas on what needs to be done ?

attached screen-shots of Ranger policies for HDFS location & key

screen-shot-2017-01-24-at-62538-pm.png

screen-shot-2017-01-24-at-62459-pm.png

----------------------------------------------------------------

hbase(main):003:0> list TABLE ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2314) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:853) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:53136) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Here is some help for this command: List all tables in hbase. Optional regular expression parameter could be used to filter the output. Examples: hbase> list hbase> list 'abc.*' hbase> list 'ns:abc.*' hbase> list 'ns:.*'

[#2]

@Karan Alang

Something is missing. Your HMaster log is pointing to following location but

hbase.rootdir=hdfs://sandbox.hortonworks.com:8020/encrypt_hbase2/hbase/data

your hbase-site.xml points to following

<property>
      <name>hbase.rootdir</name>
      <value>hdfs://sandbox.hortonworks.com:8020/encrypt_hbase1/hbase/data</value>
    </property>
    

Let's try this. shut down everything.

hadoop fs -rm -r /encrypt_hbase1/hbase/data/* //should have no affect as this should be empty
echo "rmr /hbase-unsecure" | zookeeper-client // this should cleanup everything in zookeeper.

Then only start zookeeper. Make sure it's green in Ambari. Then start only HMaster and no need to restart region servers until HMaster is successfully started.

[#3]

@mqureshi - i created new encryption zone, /encrypt_hbase2/hbase to re-test it.

The current hbase-site.xml also points to new encryption zone -> /encrypt_hbase2/hbase

sorry, forgot to mention that earlier.

[#4]

but after you did this, did you refresh the zookeeper?

[#5]

@mqureshi - yes, i did ..

Infact, just did a redo .. 1) rmr /hbase-unsecure 2) restarted zookeeper 3) restarted hbase master

if you see the highlighted portion in hbase master log - seems it is trying to look for

/hbase-unsecure/rs/sandbox.hortonworks.com,16000,1485470635621 already deleted, retry=false

which seems to be not getting created ?

Zookeeper client ->

[zk: sandbox.hortonworks.com:2181(CONNECTED) 3] ls /hbase-unsecure [recovering-regions, splitWAL, rs, backup-masters, region-in-transition, draining, table, table-lock] [zk: sandbox.hortonworks.com:2181(CONNECTED) 4] ls /hbase-unsecure/rs []

HBase Master logs ->

-----------------------------------------------

[root@sandbox ~]# tail -f /var/log/hbase/hbase-hbase-master-sandbox.hortonworks.com.log 2017-01-26 22:44:01,359 INFO [master/sandbox.hortonworks.com/10.0.2.15:16000-EventThread] zookeeper.ClientCnxn: EventThread shut down 2017-01-26 22:44:01,359 INFO [master/sandbox.hortonworks.com/10.0.2.15:16000] zookeeper.ZooKeeper: Session: 0x159dcf27e800001 closed 2017-01-26 22:44:01,370 INFO [master/sandbox.hortonworks.com/10.0.2.15:16000] regionserver.HRegionServer: stopping server sandbox.hortonworks.com,16000,1485470635621; all regions closed. 2017-01-26 22:44:01,371 INFO [master/sandbox.hortonworks.com/10.0.2.15:16000] hbase.ChoreService: Chore service for: sandbox.hortonworks.com,16000,1485470635621 had [] on shutdown 2017-01-26 22:44:01,379 INFO [master/sandbox.hortonworks.com/10.0.2.15:16000] ipc.RpcServer: Stopping server on 16000 2017-01-26 22:44:01,382 INFO [master/sandbox.hortonworks.com/10.0.2.15:16000] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/rs/sandbox.hortonworks.com,16000,1485470635621 already deleted, retry=false 2017-01-26 22:44:01,384 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2017-01-26 22:44:01,385 INFO [master/sandbox.hortonworks.com/10.0.2.15:16000] zookeeper.ZooKeeper: Session: 0x159dcf27e800000 closed 2017-01-26 22:44:01,385 INFO [master/sandbox.hortonworks.com/10.0.2.15:16000] regionserver.HRegionServer: stopping server sandbox.hortonworks.com,16000,1485470635621; zookeeper connection closed. 2017-01-26 22:44:01,385 INFO [master/sandbox.hortonworks.com/10.0.2.15:16000] regionserver.HRegionServer: master/sandbox.hortonworks.com/10.0.2.15:16000 exiting

[#6 · Accepted solution]

@mqureshi -

Appreciate your help in this. .. finally figured out the cause of the issue.

I was trying to use Ranger to restrict access to tables created in Encryption Zone, and in the process had removed access to user 'hbase' - to the encrypted HDFS location, and the key.

This was causing the issue in starting up HBase... I've added back the permisson to the HDFS location & the key, and am able to startup HBase, create & access tables.

However, one more issue -

How do i restrict access to the table created (using Ranger)

Here is what i did -

1) Removed Global access to Hbase tables 2) Gave access to table created - 'emp'

However, now i'm not able to see the table created.

Any ideas on how to achieve this ?

[#7]

@mqureshi -

i had a to do a - su hbase, and then launch hbase shell .. else the user root was trying to access the hbase table.

i'm able to create & access the table now. Thnx.