About josr89

josr89 · ‎01-11-2024

We are running a HDP Cloudera Cluster version 3.1, kerberized. All the services but HBase are starting with no issue. For HBase, when the service starts, both nodes are starting as Master, then switch to Standby, and finally goes down. telling in the logs that the service doesn´t have owner permissions in an specific HDFS filesystem path, for an user that is not the one starting the service. Even further, eventhough the HDFS before I start the service, at the mentioned filesystem path has everything owned by an user with a name "userA" and a group "groupA", all of the sudden when starting, it changes, for instance, the .tmp directory to be owned by a different user, let´s call it "userB" In Ranger the rules are properly defined, Kerberos is working fine (tickets are valid), and it worth to mention that it started all of the sudden. Herewith the error stack trace (or part of it): 2024-01-11 13:01:05,695 DEBUG [Thread-18] util.FSTableDescriptors: Current path=hdfs://env1/apps/hbase/data/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2024-01-11 13:01:05,799 ERROR [Thread-18] master.MasterFileSystem: Failed to create or set permission on staging directory hdfs://env1/apps/hbase/data/staging 2024-01-11 13:01:05,805 ERROR [Thread-18] master.HMaster: Failed to become active master java.io.IOException: Failed to create or set permission on staging directory hdfs://env1/apps/hbase/data/staging at org.apache.hadoop.hbase.master.MasterFileSystem.checkStagingDir(MasterFileSystem.java:381) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:169) at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:122) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:827) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2225) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:568) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied. user=userA is not the owner of inode=/apps/hbase/data/staging at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:302) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:269) at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:58 Any idea would be appreciated. I have tried with plenty of forums entries I have found. Have a great day!

josr89 · ‎08-30-2023

We have a cloudera cluster version 3.1, fully kerberized. We have ranger with HDFS, Hive, HBASE, among other plugins active. All of the sudden, the HBASE service gets stopped after saying that the Ambari Metrics user does not have permissions under /apps/hbase/*. We have the ranger policy in HDFS where the user has read, write, execute permissions in the mentioned paths. Yet we see in the Ranger audit page the denial, but after HDP ACL rule, not Ranger rule. I did the plugin deactivation and activation, I checked as well the plugins sync time, and the rules in place @ the hbase server servers (all looks ok) We have another cluster with same configuration, that is working perfectly fine. Any idea, where to look at? I have the feeling that the problem is coming with Kerberos, but I don´t see any "evident" issue. Thanks!

josr89 · ‎08-01-2023

Hi @Kartik_Agarwal ! I solved this a couple of weeks ago, but you are right with your comments: In my case the file was having the right permission set, but the parent directory had not. So with the change of permissions to the parent directory, it worked fine. Thanks for your support!

josr89 · ‎07-03-2023

Hello team, I have tried to change the cluster certificates following the indications from the manual, plus internal company documents. I am stucked when starting the cluster, as when I try to start the INFR Solr instance, it returns a "no valid keystore" message. I have tried with the passwords that are configured in the config files for the service, and it works smoothly. I have done manual comparisson to working environments and everything seems to be in order. Any idea where to hit? Thanks in advance 🙂

josr89 · ‎03-09-2023

Hi, I have inherited some Hadoop 3.1.1 clusters. There is a configured connection between SAP BO and Hadoop using Knox 1.0.0 as the Proxy for authentication. The problem that I have is that I don´t count with user to authenticate to Knox, and I need to edit the gateway. Is there a way to recover the admin password? Thanks for your support.

josr89 · ‎09-08-2022

Hi Jim, thanks for your prompt answer. Here my answers: 1) what do the logs say? nothing relevant, just the illegal argument exception, for example: 2022-09-08 16:15:42,386 WARN server.AuthenticationFilter (AuthenticationFilter.java:doFilter(608)) - Authentication exception: java.lang.IllegalArgumentException 2)Thanks for the information provided, but I am not able to identify if the config is ok or not. One important thing perhaps to mention is that the Yarn Server is in one domain (let´s call it A) but the user authentication is done to a different domain (let´s call it B). For what I see in the headers, the server side and the client side are making reference of domain A, which makes sense, but I can´t identify in which moment the illegal argument comes up 3) the domain controllers were deprecated, and we had to reconfigure the Kerberos information by the servers in the Cluster plus run the Ambari Ldap Setup to make reference to the new DC. We did not modify anything regarding Yarn, Oozie or the other services. Thanks for your help, Jesus

josr89 · ‎09-08-2022

Hello community, After a recent Domain Controler update, our fully kerberized Cloudera cluster 3.1.1 does not allow us to login the web interfaces of Oozie and YARN (the underneath services are working fine, apps are up and running) When authenticating, we receive in the browser the following message: HTTP 403 - Authentication exception: java.lang.IllegalArgumentException. We have tried with different browsers, and configured them to work with the Kerberos authentication as explained in other threads. It is important to mention, that the authentication worked fine before the Domain Controller update. Any ideas on how to solve this? thanks for your support.

Online	Offline
Last Visited	‎02-07-2024 11:33 AM

Member Since	‎09-08-2022 01:26 AM
Last Visited	‎02-07-2024 11:33 AM
Posts	7
Kudos received	3

Cloudera Community

HBASE Master initialization issue

HDFS Ranger plugin policies are not working as exp...

Re: AMBARI Infra SLR does not start the instance a...

AMBARI Infra SLR does not start the instance after...

Recover Knox admin password

Re: OOZIE / YARN WEB UI do not authenticate - HTTP...

OOZIE / YARN WEB UI do not authenticate - HTTP 403...