Reply
Expert Contributor
Posts: 128
Registered: ‎08-08-2013
Accepted Solution

how to remove a node in zookeeper, forcibly ?

Hi,

 

after disabling Kerberos the HBase Master won't start because no access to zookeeper znode /hbase/shutdown. I tried to remove it in zookeeper shell (started as user root), but no success =>

 

[zk: localhost:2181(CONNECTED) 3] rmr /hbase/shutdown
Authentication is not valid : /hbase/shutdown
[zk: localhost:2181(CONNECTED) 4] getAcl /hbase/shutdown
'sasl,'hbase
: cdrwa
[zk: localhost:2181(CONNECTED) 5]

 

 

How can I forcibly deltete that subtree to be able to start HBase afterwards?

 

Error in HBase Master log:

 

2014-03-31 10:23:41,760 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x4451714a72b004b Unable to get data of znode /hbase/shutdown
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/shutdown

 

thanks in advance...Gerd...

 

Posts: 416
Topics: 51
Kudos: 73
Solutions: 48
Registered: ‎06-26-2013

Re: how to remove a node in zookeeper, forcibly ?

Gerd,

 

  backing out kerberos is not an automatic process currently as there can be many services using Zookeeper and it retains those ACLs which were set while kerberos was enabled.  We have developed a little java program for our customers that backs out the ACLs from ZK, but all it really does is iterate over all the znodes in /hbase and set their acls to world:anyone.

 

So, you can just manually do this as well.  This is an example:

 

setAcl /hbase world:anyone:cdrwa

You would need to do that on every znode under /hbase and the master will start.

 

HTH,

 

Clint

 

Expert Contributor
Posts: 128
Registered: ‎08-08-2013

Re: how to remove a node in zookeeper, forcibly ?

Hi Clint,

 

many thanks, the "world:anyone" combination was the missing fact ;)

 

Despite a "Authentication is not valid" message at executing the setAcl I was able to get access/delete the "shutdown" node under /hbase

 

Log:

 

[zk: localhost:2181(CONNECTED) 4] getAcl /hbase/shutdown
'sasl,'hbase
: cdrwa
[zk: localhost:2181(CONNECTED) 5] setAcl /hbase/shutdown world:anyone:cdrwa
Authentication is not valid : /hbase/shutdown
[zk: localhost:2181(CONNECTED) 6] delete /hbase/shutdown
[zk: localhost:2181(CONNECTED) 7] getAcl /hbase/shutdown
Node does not exist: /hbase/shutdown

 

HBase is up and running again, that's what matters :D

 

regards....: Gerd :...

New Contributor
Posts: 1
Registered: ‎09-02-2014

Re: how to remove a node in zookeeper, forcibly ?

still not able to delete 

 

zk: localhost:2181(CONNECTED) 13] setAcl /hbase world:anyone:cdrwa              

Authentication is not valid : /hbase

[zk: localhost:2181(CONNECTED) 14] setAcl /hbase/shutdown world:anyone:cdrwa

Authentication is not valid : /hbase/shutdown

zk: localhost:2181(CONNECTED) 16] rmr /hbase/shutdown   

Authentication is not valid : /hbase/shutdown

[zk: localhost:2181(CONNECTED) 15] delete /hbase/shutdown

Authentication is not valid : /hbase/shutdown

New Contributor
Posts: 2
Registered: ‎11-12-2015

Re: how to remove a node in zookeeper, forcibly ?

I am having the same issue. I cannot sent the ACLs and also cannot delete the ACLs because they were created by the kerberized environment. Is there any way to work around this?

Posts: 1,483
Kudos: 241
Solutions: 225
Registered: ‎07-31-2013

Re: how to remove a node in zookeeper, forcibly ?

The simplest way is to disable ACLs in ZK, and restarting ZK.

Append the below Java system property to CM -> Zookeeper -> Configuration -> "ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)" field:

-Dzookeeper.skipACL=true

Save and restart ZK. Run your rmr command.

You can revert that change back and re-restart if you want ACLs feature kept after.

Another way is to declare a ZK superuser digest and using that whenever you face this (i.e. whenever you've lost the identity or mechanism of authentication to the ACL'd znode). This is documented at http://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_authOptions, and the option can be similarly added.
Backline Customer Operations Engineer
Posts: 1,483
Kudos: 241
Solutions: 225
Registered: ‎07-31-2013

Re: how to remove a node in zookeeper, forcibly ?

Just a correction to my typo above, the right config field for the skipACL switch is not the "Environment Advanced Configuration Snippet" but is instead is "Java Configuration Options for Zookeeper Server" in CM -> ZooKeeper -> Configuration page.
Backline Customer Operations Engineer
Expert Contributor
Posts: 81
Registered: ‎07-01-2015

Re: how to remove a node in zookeeper, forcibly ?

Hi Harsh,

 I had the same issue, put the -Dzookeeper.skipACL=true option to Java Configuration Options for Zookeeper Server restarted the Zookeeper service, but still having this error message:

 

[zk: localhost:2181(CONNECTED) 0] rmr /hbase
Authentication is not valid : /hbase/backup-masters

 

Tomas

Explorer
Posts: 19
Registered: ‎01-25-2016

Re: how to remove a node in zookeeper, forcibly ?

Hi Harsh,

 

Your option didnt work for me as well. I am facing similar issue, can you please help?

New Contributor
Posts: 1
Registered: ‎03-29-2017

Re: how to remove a node in zookeeper, forcibly ?

http://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos-on-Cloudera-EXpres.... the url had a good answer with Mr Ben, i had tried by myself, it is valid.
Announcements