HBASE Master initialization issue

[#1]

We are running a HDP Cloudera Cluster version 3.1, kerberized.

All the services but HBase are starting with no issue. For HBase, when the service starts, both nodes are starting as Master, then switch to Standby, and finally goes down. telling in the logs that the service doesn´t have owner permissions in an specific HDFS filesystem path, for an user that is not the one starting the service.

Even further, eventhough the HDFS before I start the service, at the mentioned filesystem path has everything owned by an user with a name "userA" and a group "groupA", all of the sudden when starting, it changes, for instance, the .tmp directory to be owned by a different user, let´s call it "userB"

In Ranger the rules are properly defined, Kerberos is working fine (tickets are valid), and it worth to mention that it started all of the sudden.

Herewith the error stack trace (or part of it):

2024-01-11 13:01:05,695 DEBUG [Thread-18] util.FSTableDescriptors: Current path=hdfs://env1/apps/hbase/data/data/hbase/meta/.tabledesc/.tableinfo.0000000001

2024-01-11 13:01:05,799 ERROR [Thread-18] master.MasterFileSystem: Failed to create or set permission on staging directory hdfs://env1/apps/hbase/data/staging

2024-01-11 13:01:05,805 ERROR [Thread-18] master.HMaster: Failed to become active master

java.io.IOException: Failed to create or set permission on staging directory hdfs://env1/apps/hbase/data/staging

        at org.apache.hadoop.hbase.master.MasterFileSystem.checkStagingDir(MasterFileSystem.java:381)

        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:169)

        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:122)

        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:827)

        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2225)

        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:568)

        at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.hadoop.security.AccessControlException: Permission denied. user=userA is not the owner of inode=/apps/hbase/data/staging

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:302)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:269)

        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:58

Any idea would be appreciated. I have tried with plenty of forums entries I have found.

Have a great day!

[#2 · Accepted solution]

@josr89 seems like you have permission issues for below path

hdfs://env1/apps/hbase/data/staging

Kindly assign proper permission for the same path for userA from the Ranger UI

[#3 · Accepted solution]

@josr89 seems like you have permission issues for below path

hdfs://env1/apps/hbase/data/staging

Kindly assign proper permission for the same path for userA from the Ranger UI

[#4 · Accepted solution]

Hi @josr89 you can provide WRITE access to user  "userA" to the below path under the "cm_hdfs" repository in Ranger, wait for plugins to sync and then rerun the operation.

path: /apps/hbase/data/staging