We are running a HDP Cloudera Cluster version 3.1, kerberized.
All the services but HBase are starting with no issue. For HBase, when the service starts, both nodes are starting as Master, then switch to Standby, and finally goes down. telling in the logs that the service doesn´t have owner permissions in an specific HDFS filesystem path, for an user that is not the one starting the service.
Even further, eventhough the HDFS before I start the service, at the mentioned filesystem path has everything owned by an user with a name "userA" and a group "groupA", all of the sudden when starting, it changes, for instance, the .tmp directory to be owned by a different user, let´s call it "userB"
In Ranger the rules are properly defined, Kerberos is working fine (tickets are valid), and it worth to mention that it started all of the sudden.
Herewith the error stack trace (or part of it):
2024-01-11 13:01:05,695 DEBUG [Thread-18] util.FSTableDescriptors: Current path=hdfs://env1/apps/hbase/data/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2024-01-11 13:01:05,799 ERROR [Thread-18] master.MasterFileSystem: Failed to create or set permission on staging directory hdfs://env1/apps/hbase/data/staging
2024-01-11 13:01:05,805 ERROR [Thread-18] master.HMaster: Failed to become active master
java.io.IOException: Failed to create or set permission on staging directory hdfs://env1/apps/hbase/data/staging
at org.apache.hadoop.hbase.master.MasterFileSystem.checkStagingDir(MasterFileSystem.java:381)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:169)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:122)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:827)
at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2225)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:568)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.security.AccessControlException: Permission denied. user=userA is not the owner of inode=/apps/hbase/data/staging
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:302)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:269)
at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:58
Any idea would be appreciated. I have tried with plenty of forums entries I have found.
Have a great day!
