Skip to main content
josr89
Silver ⭐️⭐️
January 11, 2024
Solved

HBASE Master initialization issue

We are running a HDP Cloudera Cluster version 3.1, kerberized.

All the services but HBase are starting with no issue. For HBase, when the service starts, both nodes are starting as Master, then switch to Standby, and finally goes down. telling in the logs that the service doesn´t have owner permissions in an specific HDFS filesystem path, for an user that is not the one starting the service.

 

Even further, eventhough the HDFS before I start the service, at the mentioned filesystem path has everything owned by an user with a name "userA" and a group "groupA", all of the sudden when starting, it changes, for instance, the .tmp directory to be owned by a different user, let´s call it "userB"

 

In Ranger the rules are properly defined, Kerberos is working fine (tickets are valid), and it worth to mention that it started all of the sudden.

Herewith the error stack trace (or part of it):

 

2024-01-11 13:01:05,695 DEBUG [Thread-18] util.FSTableDescriptors: Current path=hdfs://env1/apps/hbase/data/data/hbase/meta/.tabledesc/.tableinfo.0000000001

2024-01-11 13:01:05,799 ERROR [Thread-18] master.MasterFileSystem: Failed to create or set permission on staging directory hdfs://env1/apps/hbase/data/staging

2024-01-11 13:01:05,805 ERROR [Thread-18] master.HMaster: Failed to become active master

java.io.IOException: Failed to create or set permission on staging directory hdfs://env1/apps/hbase/data/staging

at org.apache.hadoop.hbase.master.MasterFileSystem.checkStagingDir(MasterFileSystem.java:381)

at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:169)

at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:122)

at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:827)

at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2225)

at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:568)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.hadoop.security.AccessControlException: Permission denied. user=userA is not the owner of inode=/apps/hbase/data/staging

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:302)

at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:269)

at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:58

 

Any idea would be appreciated. I have tried with plenty of forums entries I have found.

 

Have a great day!

 

9een
9eenAnswer
Gold ⭐️
March 4, 2024

@josr89 seems like you have permission issues for below path

hdfs://env1/apps/hbase/data/staging

Kindly assign proper permission for the same path for userA from the Ranger UI

2 replies

9een
Gold ⭐️
9eenAnswer
Gold ⭐️
March 4, 2024

@josr89 seems like you have permission issues for below path

hdfs://env1/apps/hbase/data/staging

Kindly assign proper permission for the same path for userA from the Ranger UI

Translate
Rajat_710
Gold ⭐️⭐️
Gold ⭐️⭐️
March 21, 2024

Hi @josr89 you can provide WRITE access to user  "userA" to the below path under the "cm_hdfs" repository in Ranger, wait for plugins to sync and then rerun the operation.

path: /apps/hbase/data/staging

 

Translate