Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to remove a node in zookeeper, forcibly ?

SOLVED Go to solution

how to remove a node in zookeeper, forcibly ?

Guru

Hi,

 

after disabling Kerberos the HBase Master won't start because no access to zookeeper znode /hbase/shutdown. I tried to remove it in zookeeper shell (started as user root), but no success =>

 

[zk: localhost:2181(CONNECTED) 3] rmr /hbase/shutdown
Authentication is not valid : /hbase/shutdown
[zk: localhost:2181(CONNECTED) 4] getAcl /hbase/shutdown
'sasl,'hbase
: cdrwa
[zk: localhost:2181(CONNECTED) 5]

 

 

How can I forcibly deltete that subtree to be able to start HBase afterwards?

 

Error in HBase Master log:

 

2014-03-31 10:23:41,760 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x4451714a72b004b Unable to get data of znode /hbase/shutdown
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/shutdown

 

thanks in advance...Gerd...

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: how to remove a node in zookeeper, forcibly ?

Master Collaborator

Gerd,

 

  backing out kerberos is not an automatic process currently as there can be many services using Zookeeper and it retains those ACLs which were set while kerberos was enabled.  We have developed a little java program for our customers that backs out the ACLs from ZK, but all it really does is iterate over all the znodes in /hbase and set their acls to world:anyone.

 

So, you can just manually do this as well.  This is an example:

 

setAcl /hbase world:anyone:cdrwa

You would need to do that on every znode under /hbase and the master will start.

 

HTH,

 

Clint

 

11 REPLIES 11

Re: how to remove a node in zookeeper, forcibly ?

Master Collaborator

Gerd,

 

  backing out kerberos is not an automatic process currently as there can be many services using Zookeeper and it retains those ACLs which were set while kerberos was enabled.  We have developed a little java program for our customers that backs out the ACLs from ZK, but all it really does is iterate over all the znodes in /hbase and set their acls to world:anyone.

 

So, you can just manually do this as well.  This is an example:

 

setAcl /hbase world:anyone:cdrwa

You would need to do that on every znode under /hbase and the master will start.

 

HTH,

 

Clint

 

Re: how to remove a node in zookeeper, forcibly ?

Guru

Hi Clint,

 

many thanks, the "world:anyone" combination was the missing fact ;)

 

Despite a "Authentication is not valid" message at executing the setAcl I was able to get access/delete the "shutdown" node under /hbase

 

Log:

 

[zk: localhost:2181(CONNECTED) 4] getAcl /hbase/shutdown
'sasl,'hbase
: cdrwa
[zk: localhost:2181(CONNECTED) 5] setAcl /hbase/shutdown world:anyone:cdrwa
Authentication is not valid : /hbase/shutdown
[zk: localhost:2181(CONNECTED) 6] delete /hbase/shutdown
[zk: localhost:2181(CONNECTED) 7] getAcl /hbase/shutdown
Node does not exist: /hbase/shutdown

 

HBase is up and running again, that's what matters :D

 

regards....: Gerd :...

Re: how to remove a node in zookeeper, forcibly ?

New Contributor

still not able to delete 

 

zk: localhost:2181(CONNECTED) 13] setAcl /hbase world:anyone:cdrwa              

Authentication is not valid : /hbase

[zk: localhost:2181(CONNECTED) 14] setAcl /hbase/shutdown world:anyone:cdrwa

Authentication is not valid : /hbase/shutdown

zk: localhost:2181(CONNECTED) 16] rmr /hbase/shutdown   

Authentication is not valid : /hbase/shutdown

[zk: localhost:2181(CONNECTED) 15] delete /hbase/shutdown

Authentication is not valid : /hbase/shutdown

Re: how to remove a node in zookeeper, forcibly ?

Explorer

I am having the same issue. I cannot sent the ACLs and also cannot delete the ACLs because they were created by the kerberized environment. Is there any way to work around this?

Re: how to remove a node in zookeeper, forcibly ?

Master Guru
The simplest way is to disable ACLs in ZK, and restarting ZK.

Append the below Java system property to CM -> Zookeeper -> Configuration -> "ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)" field:

-Dzookeeper.skipACL=true

Save and restart ZK. Run your rmr command.

You can revert that change back and re-restart if you want ACLs feature kept after.

Another way is to declare a ZK superuser digest and using that whenever you face this (i.e. whenever you've lost the identity or mechanism of authentication to the ACL'd znode). This is documented at http://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#sc_authOptions, and the option can be similarly added.

Re: how to remove a node in zookeeper, forcibly ?

Master Guru
Just a correction to my typo above, the right config field for the skipACL switch is not the "Environment Advanced Configuration Snippet" but is instead is "Java Configuration Options for Zookeeper Server" in CM -> ZooKeeper -> Configuration page.

Re: how to remove a node in zookeeper, forcibly ?

Hi Harsh,

 

Your option didnt work for me as well. I am facing similar issue, can you please help?

Re: how to remove a node in zookeeper, forcibly ?

New Contributor
http://community.cloudera.com/t5/Cloudera-Manager-Installation/Disabling-Kerberos-on-Cloudera-EXpres.... the url had a good answer with Mr Ben, i had tried by myself, it is valid.
Highlighted

Re: how to remove a node in zookeeper, forcibly ?

Master Collaborator

Hi Harsh,

 I had the same issue, put the -Dzookeeper.skipACL=true option to Java Configuration Options for Zookeeper Server restarted the Zookeeper service, but still having this error message:

 

[zk: localhost:2181(CONNECTED) 0] rmr /hbase
Authentication is not valid : /hbase/backup-masters

 

Tomas