question Re: how to remove a node in zookeeper, forcibly ? in Support Questions

how to remove a node in zookeeper, forcibly ?

geko — Fri, 16 Sep 2022 08:56:19 GMT

Hi,

after disabling Kerberos the HBase Master won't start because no access to zookeeper znode /hbase/shutdown. I tried to remove it in zookeeper shell (started as user root), but no success =>

[zk: localhost:2181(CONNECTED) 3] rmr /hbase/shutdown
Authentication is not valid : /hbase/shutdown
[zk: localhost:2181(CONNECTED) 4] getAcl /hbase/shutdown
'sasl,'hbase
: cdrwa
[zk: localhost:2181(CONNECTED) 5]

How can I forcibly deltete that subtree to be able to start HBase afterwards?

Error in HBase Master log:

2014-03-31 10:23:41,760 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x4451714a72b004b Unable to get data of znode /hbase/shutdown
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/shutdown

thanks in advance...Gerd...

Re: how to remove a node in zookeeper, forcibly ?

Clint — Tue, 01 Apr 2014 21:54:25 GMT

Gerd,

backing out kerberos is not an automatic process currently as there can be many services using Zookeeper and it retains those ACLs which were set while kerberos was enabled. We have developed a little java program for our customers that backs out the ACLs from ZK, but all it really does is iterate over all the znodes in /hbase and set their acls to world:anyone.

So, you can just manually do this as well. This is an example:

setAcl /hbase world:anyone:cdrwa

You would need to do that on every znode under /hbase and the master will start.

HTH,

Clint

Re: how to remove a node in zookeeper, forcibly ?

geko — Wed, 02 Apr 2014 07:14:44 GMT

Hi Clint,

many thanks, the "world:anyone" combination was the missing fact 😉

Despite a "Authentication is not valid" message at executing the setAcl I was able to get access/delete the "shutdown" node under /hbase

Log:

[zk: localhost:2181(CONNECTED) 4] getAcl /hbase/shutdown
'sasl,'hbase
: cdrwa
[zk: localhost:2181(CONNECTED) 5] setAcl /hbase/shutdown world:anyone:cdrwa
Authentication is not valid : /hbase/shutdown
[zk: localhost:2181(CONNECTED) 6] delete /hbase/shutdown
[zk: localhost:2181(CONNECTED) 7] getAcl /hbase/shutdown
Node does not exist: /hbase/shutdown

HBase is up and running again, that's what matters 😄

regards....: Gerd :...

Re: how to remove a node in zookeeper, forcibly ?

lessc0de — Tue, 02 Sep 2014 21:51:38 GMT

still not able to delete

zk: localhost:2181(CONNECTED) 13] setAcl /hbase world:anyone:cdrwa

Authentication is not valid : /hbase

[zk: localhost:2181(CONNECTED) 14] setAcl /hbase/shutdown world:anyone:cdrwa

Authentication is not valid : /hbase/shutdown

zk: localhost:2181(CONNECTED) 16] rmr /hbase/shutdown

Authentication is not valid : /hbase/shutdown

[zk: localhost:2181(CONNECTED) 15] delete /hbase/shutdown

Authentication is not valid : /hbase/shutdown

Re: how to remove a node in zookeeper, forcibly ?

akafazov — Wed, 08 Jun 2016 16:08:24 GMT

I am having the same issue. I cannot sent the ACLs and also cannot delete the ACLs because they were created by the kerberized environment. Is there any way to work around this?

Re: how to remove a node in zookeeper, forcibly ?

Harsh J — Thu, 09 Jun 2016 07:35:38 GMT

The simplest way is to disable ACLs in ZK, and restarting ZK.

Append the below Java system property to CM -> Zookeeper -> Configuration -> "ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)" field:

-Dzookeeper.skipACL=true

Save and restart ZK. Run your rmr command.

You can revert that change back and re-restart if you want ACLs feature kept after.

Another way is to declare a ZK superuser digest and using that whenever you face this (i.e. whenever you've lost the identity or mechanism of authentication to the ACL'd znode). This is documented at http://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_authOptions, and the option can be similarly added.

Re: how to remove a node in zookeeper, forcibly ?

Harsh J — Thu, 18 Aug 2016 02:37:39 GMT

Just a correction to my typo above, the right config field for the skipACL switch is not the "Environment Advanced Configuration Snippet" but is instead is "Java Configuration Options for Zookeeper Server" in CM -> ZooKeeper -> Configuration page.

Re: how to remove a node in zookeeper, forcibly ?

Tomas79 — Tue, 22 Nov 2016 07:58:13 GMT

Hi Harsh,

I had the same issue, put the -Dzookeeper.skipACL=true option to Java Configuration Options for Zookeeper Server restarted the Zookeeper service, but still having this error message:

[zk: localhost:2181(CONNECTED) 0] rmr /hbase
Authentication is not valid : /hbase/backup-masters

Tomas

Re: how to remove a node in zookeeper, forcibly ?

SiddeshSamarth — Mon, 13 Mar 2017 13:08:03 GMT

Hi Harsh,

Your option didnt work for me as well. I am facing similar issue, can you please help?

Re: how to remove a node in zookeeper, forcibly ?

joyken — Thu, 30 Mar 2017 01:09:33 GMT

http://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos-on-Cloudera-EXpress-5-5-1-HBase-issue/m-p/42482/highlight/true#M7622. the url had a good answer with Mr Ben, i had tried by myself, it is valid.

Re: how to remove a node in zookeeper, forcibly ?

ganeshkumarj — Wed, 26 Apr 2017 09:57:24 GMT

It should be -Dzookeeper.skipACL=yes (Be careful not true,yes). And it works for me

Re: how to remove a node in zookeeper, forcibly ?

Ayush — Sat, 13 Oct 2018 10:54:09 GMT

Hi Harsh,

Thanks alot for your support.

Really appreciate.

I was able to make hbase stable by adding the line mentioned by you but the only one change was reuiqred.

-Dzookeeper.skipACL=yes

we need to give "yes" not true.

It worked for me.

Thanks for making my cluster happpy.

Regards

Ayush