how to remove a node in zookeeper, forcibly ?

[#1]

Hi,

after disabling Kerberos the HBase Master won't start because no access to zookeeper znode /hbase/shutdown. I tried to remove it in zookeeper shell (started as user root), but no success =>

[zk: localhost:2181(CONNECTED) 3] rmr /hbase/shutdown
Authentication is not valid : /hbase/shutdown
[zk: localhost:2181(CONNECTED) 4] getAcl /hbase/shutdown
'sasl,'hbase
: cdrwa
[zk: localhost:2181(CONNECTED) 5]

How can I forcibly deltete that subtree to be able to start HBase afterwards?

Error in HBase Master log:

2014-03-31 10:23:41,760 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x4451714a72b004b Unable to get data of znode /hbase/shutdown
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/shutdown

thanks in advance...Gerd...

[#2 · Accepted solution]

Gerd,

  backing out kerberos is not an automatic process currently as there can be many services using Zookeeper and it retains those ACLs which were set while kerberos was enabled.  We have developed a little java program for our customers that backs out the ACLs from ZK, but all it really does is iterate over all the znodes in /hbase and set their acls to world:anyone.

So, you can just manually do this as well.  This is an example:

setAcl /hbase world:anyone:cdrwa

You would need to do that on every znode under /hbase and the master will start.

HTH,

Clint

[#3]

Hi Clint,

many thanks, the "world:anyone" combination was the missing fact 😉

Despite a "Authentication is not valid" message at executing the setAcl I was able to get access/delete the "shutdown" node under /hbase

Log:

[zk: localhost:2181(CONNECTED) 4] getAcl /hbase/shutdown
'sasl,'hbase
: cdrwa
[zk: localhost:2181(CONNECTED) 5] setAcl /hbase/shutdown world:anyone:cdrwa
Authentication is not valid : /hbase/shutdown
[zk: localhost:2181(CONNECTED) 6] delete /hbase/shutdown
[zk: localhost:2181(CONNECTED) 7] getAcl /hbase/shutdown
Node does not exist: /hbase/shutdown

HBase is up and running again, that's what matters 😄

regards....: Gerd :...

[#4]

still not able to delete 

zk: localhost:2181(CONNECTED) 13] setAcl /hbase world:anyone:cdrwa              

Authentication is not valid : /hbase

[zk: localhost:2181(CONNECTED) 14] setAcl /hbase/shutdown world:anyone:cdrwa

Authentication is not valid : /hbase/shutdown

zk: localhost:2181(CONNECTED) 16] rmr /hbase/shutdown   

Authentication is not valid : /hbase/shutdown

[zk: localhost:2181(CONNECTED) 15] delete /hbase/shutdown

Authentication is not valid : /hbase/shutdown

[#5]

I am having the same issue. I cannot sent the ACLs and also cannot delete the ACLs because they were created by the kerberized environment. Is there any way to work around this?

[#6]

The simplest way is to disable ACLs in ZK, and restarting ZK.

Append the below Java system property to CM -> Zookeeper -> Configuration -> "ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)" field:

-Dzookeeper.skipACL=true

Save and restart ZK. Run your rmr command.

You can revert that change back and re-restart if you want ACLs feature kept after.

Another way is to declare a ZK superuser digest and using that whenever you face this (i.e. whenever you've lost the identity or mechanism of authentication to the ACL'd znode). This is documented at http://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_authOptions, and the option can be similarly added.

[#7]

Just a correction to my typo above, the right config field for the skipACL switch is not the "Environment Advanced Configuration Snippet" but is instead is "Java Configuration Options for Zookeeper Server" in CM -> ZooKeeper -> Configuration page.

[#8]

Hi Harsh,

Your option didnt work for me as well. I am facing similar issue, can you please help?

[#9]

http://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos-on-Cloudera-EXpres.... the url had a good answer with Mr Ben, i had tried by myself, it is valid.

[#10]

Hi Harsh,

 I had the same issue, put the -Dzookeeper.skipACL=true option to Java Configuration Options for Zookeeper Server restarted the Zookeeper service, but still having this error message:

[zk: localhost:2181(CONNECTED) 0] rmr /hbase
Authentication is not valid : /hbase/backup-masters

Tomas