Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to remove a node in zookeeper, forcibly ?

avatar
Guru

Hi,

 

after disabling Kerberos the HBase Master won't start because no access to zookeeper znode /hbase/shutdown. I tried to remove it in zookeeper shell (started as user root), but no success =>

 

[zk: localhost:2181(CONNECTED) 3] rmr /hbase/shutdown
Authentication is not valid : /hbase/shutdown
[zk: localhost:2181(CONNECTED) 4] getAcl /hbase/shutdown
'sasl,'hbase
: cdrwa
[zk: localhost:2181(CONNECTED) 5]

 

 

How can I forcibly deltete that subtree to be able to start HBase afterwards?

 

Error in HBase Master log:

 

2014-03-31 10:23:41,760 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x4451714a72b004b Unable to get data of znode /hbase/shutdown
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/shutdown

 

thanks in advance...Gerd...

 

1 ACCEPTED SOLUTION

avatar
Guru

Gerd,

 

  backing out kerberos is not an automatic process currently as there can be many services using Zookeeper and it retains those ACLs which were set while kerberos was enabled.  We have developed a little java program for our customers that backs out the ACLs from ZK, but all it really does is iterate over all the znodes in /hbase and set their acls to world:anyone.

 

So, you can just manually do this as well.  This is an example:

 

setAcl /hbase world:anyone:cdrwa

You would need to do that on every znode under /hbase and the master will start.

 

HTH,

 

Clint

 

View solution in original post

11 REPLIES 11

avatar
Guru

Gerd,

 

  backing out kerberos is not an automatic process currently as there can be many services using Zookeeper and it retains those ACLs which were set while kerberos was enabled.  We have developed a little java program for our customers that backs out the ACLs from ZK, but all it really does is iterate over all the znodes in /hbase and set their acls to world:anyone.

 

So, you can just manually do this as well.  This is an example:

 

setAcl /hbase world:anyone:cdrwa

You would need to do that on every znode under /hbase and the master will start.

 

HTH,

 

Clint

 

avatar
Guru

Hi Clint,

 

many thanks, the "world:anyone" combination was the missing fact 😉

 

Despite a "Authentication is not valid" message at executing the setAcl I was able to get access/delete the "shutdown" node under /hbase

 

Log:

 

[zk: localhost:2181(CONNECTED) 4] getAcl /hbase/shutdown
'sasl,'hbase
: cdrwa
[zk: localhost:2181(CONNECTED) 5] setAcl /hbase/shutdown world:anyone:cdrwa
Authentication is not valid : /hbase/shutdown
[zk: localhost:2181(CONNECTED) 6] delete /hbase/shutdown
[zk: localhost:2181(CONNECTED) 7] getAcl /hbase/shutdown
Node does not exist: /hbase/shutdown

 

HBase is up and running again, that's what matters 😄

 

regards....: Gerd :...

avatar
New Contributor

still not able to delete 

 

zk: localhost:2181(CONNECTED) 13] setAcl /hbase world:anyone:cdrwa              

Authentication is not valid : /hbase

[zk: localhost:2181(CONNECTED) 14] setAcl /hbase/shutdown world:anyone:cdrwa

Authentication is not valid : /hbase/shutdown

zk: localhost:2181(CONNECTED) 16] rmr /hbase/shutdown   

Authentication is not valid : /hbase/shutdown

[zk: localhost:2181(CONNECTED) 15] delete /hbase/shutdown

Authentication is not valid : /hbase/shutdown

avatar
Contributor

I am having the same issue. I cannot sent the ACLs and also cannot delete the ACLs because they were created by the kerberized environment. Is there any way to work around this?

avatar
Mentor
The simplest way is to disable ACLs in ZK, and restarting ZK.

Append the below Java system property to CM -> Zookeeper -> Configuration -> "ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)" field:

-Dzookeeper.skipACL=true

Save and restart ZK. Run your rmr command.

You can revert that change back and re-restart if you want ACLs feature kept after.

Another way is to declare a ZK superuser digest and using that whenever you face this (i.e. whenever you've lost the identity or mechanism of authentication to the ACL'd znode). This is documented at http://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_authOptions, and the option can be similarly added.

avatar
Mentor
Just a correction to my typo above, the right config field for the skipACL switch is not the "Environment Advanced Configuration Snippet" but is instead is "Java Configuration Options for Zookeeper Server" in CM -> ZooKeeper -> Configuration page.

avatar

Hi Harsh,

 

Your option didnt work for me as well. I am facing similar issue, can you please help?

avatar
New Contributor
http://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos-on-Cloudera-EXpres.... the url had a good answer with Mr Ben, i had tried by myself, it is valid.

avatar

Hi Harsh,

 I had the same issue, put the -Dzookeeper.skipACL=true option to Java Configuration Options for Zookeeper Server restarted the Zookeeper service, but still having this error message:

 

[zk: localhost:2181(CONNECTED) 0] rmr /hbase
Authentication is not valid : /hbase/backup-masters

 

Tomas