Reply
Highlighted
Explorer
Posts: 20
Registered: ‎11-26-2015

Disabling Kerberos on Cloudera EXpress 5.5.1: HBase issue

Hello,

 

I have disabled the Kerberos configuration on my cluster. The hbase and Solr are not starting anymore.

 

HBASE: I have checked the 2 parameters: 

HBase Secure Authentication(hbase.security.authentication) : simple

HBase Secure Authorization(hbase.security.authorization): false

 

My master and 3 RegionServer aren't starting. I have checked the confiration generated to run the servers by the scm and the parameters are like mentionned above.

 

I have noticed anyway that the core-site.xml generated contain:

<property>
<name>hadoop.security.auth_to_local</name>
<value>RULE:[1:$1@$0](.*@\QTEST.COM\E$)s/@\QTEST.COM\E$//
RULE:[2:$1@$0](.*@\QTEST.COM\E$)s/@\QTEST.COM\E$//
DEFAULT</value>
</property>

 

Could this be the problem? Any idea is welcome, I am stuck...

 

Posts: 1,035
Topics: 1
Kudos: 258
Solutions: 128
Registered: ‎04-22-2014

Re: Disabling Kerberos on Cloudera EXpress 5.5.1: HBase issue

Hello,

 

auth_to_local is not in use if you have disabled kerberos.  I'd check to find out what the logs say in your Hbase servers to help guide the investigation.  Check them in /var/log/hbase

 

Regards,

 

Ben

Explorer
Posts: 20
Registered: ‎11-26-2015

Re: Disabling Kerberos on Cloudera EXpress 5.5.1: HBase issue

Hello,

 

I found the following meesage in the hbase logs:

 

aseZNode=/hbase Unable to get data of znode /hbase/running

 

I found a note on cancelling this from zkcli but I need to be able to start zookeeper with skipACL.

But, here I have got a new issue:

CM .> Zookeeper -> Configuration -> ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve)

I enter the following option:

-Dzookeeper.skipACL=true

 

=> I receive this error message:

ould not parse: ZooKeeper Service Environment Advanced Configuration Snippet (Safety Valve) : Could not parse parameter 'zookeeper_env_safety_valve'. Was expecting: valid variable name. Input: -Dzookeeper.skipACL=true

 

Which is weird, I don't what is wrong with this option?

Posts: 1,035
Topics: 1
Kudos: 258
Solutions: 128
Registered: ‎04-22-2014

Re: Disabling Kerberos on Cloudera EXpress 5.5.1: HBase issue

The safety valve you used is intended for OS environment variables, not java options.

Try using the following:

 

Java Configuration Options for Zookeeper Server

 

-Ben

Explorer
Posts: 20
Registered: ‎11-26-2015

Re: Disabling Kerberos on Cloudera EXpress 5.5.1: HBase issue

Hi Ben,

 

Thanks for you feedback, here is what I have done:

Zookeeper: java Configuration Options for Zookeeper Server : -Dzookeeper.skipACL=yes (Be careful not true, yes).

 

I restarted zookeeper:

I remove 2 nodes using hbase zkcli (for which I had to fix the zkcli classpath to get it running).

setAcl /hbase/running world:anyone:cdrwa

rmr /hbase/running

setAcl /hbase/backup-masters world:anyone:cdrwa

rmr /hbase/backup-masters

 

Then I restart ZK without the skipACL option.

 

Try to restart Hbase but it still crash with such a message:

2016-07-01 08:29:23,610 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2046)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2060)
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/backup-masters
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:576)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:555)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1201)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1190)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:1173)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:187)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:172)
at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:547)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:347)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2041)
... 5 more

 

 

Explorer
Posts: 20
Registered: ‎11-26-2015

Re: Disabling Kerberos on Cloudera EXpress 5.5.1: HBase issue

Hello,

 

I have applied the setAcl world:anyone:cdrwa on all items under the /hbase.

 

the HBase is starting when Zookeeper is started with -Dzookeeper.skipACL=yes

If I do not skip the ACL, the master node is not starting:

I get this message:

2016-07-01 09:02:37,158 INFO org.apache.hadoop.hbase.master.HMaster: hbase.rootdir=hdfs://nameservice1/hbase, hbase.cluster.distribute
d=true
2016-07-01 09:02:37,171 INFO org.apache.hadoop.hbase.master.HMaster: Adding backup master ZNode /hbase/backup-masters/jk1,60000,146736
3754733
2016-07-01 09:02:37,296 FATAL org.apache.hadoop.hbase.master.HMaster: Received an unexpected KeeperException, aborting
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:576)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:555)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createEphemeralNodeAndWatch(ZKUtil.java:1017)
at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.setMasterAddress(MasterAddressTracker.java:211)
at org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:164)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1480)
at java.lang.Thread.run(Thread.java:745)
2016-07-01 09:02:37,341 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Received an unexpected KeeperException, abor
ting

 

this is strange becausefrom zkcli I can't see the /hbase/master znode.

 

Any idea?

 

 

Posts: 1,035
Topics: 1
Kudos: 258
Solutions: 128
Registered: ‎04-22-2014

Re: Disabling Kerberos on Cloudera EXpress 5.5.1: HBase issue

I believe you need to delete the znodes as such

 

$ hbase zkcli
> rmr /hbase
> quit

Howerver, since you have already disabled kerberos, I believe you will need to follow steps similar to this:

 

  1. Reset the super digest by adding a temporary configuration. Using Cloudera Manager, navigate to, Zookeeper service -> Configuration choose, "Java Configuration Options for Zookeeper Server" and add the following:
-Dzookeeper.DigestAuthenticationProvider.superDigest=super:cY+9eK20soteVC3fQ83SXDvwlP0=

 

 

  1. Restart zookeeper service.
  2. Open the Zookeeper shell with the following:
    zookeeper-client -server <zk-host>
  3. Delete the znodes with:
    > addauth digest super:cloudera
    > rmr /hbase
    > quit
  4. Stop Zookeeper
  5. Remove the temp super digest added in step 1
  6. Start Zookeeper
  7. If Zookeeper starts up, try starting HBase

 NOTE:  This message board software seems to have gotten funky with the ordered list above, but I'm certain I'll cause more damage if I try to fix it.  Follow both step 1s in the order they are listed.

 

-Ben

 

Posts: 1,035
Topics: 1
Kudos: 258
Solutions: 128
Registered: ‎04-22-2014

Re: Disabling Kerberos on Cloudera EXpress 5.5.1: HBase issue

I should have noted that you should shut down HBase while doing those steps.

Explorer
Posts: 20
Registered: ‎11-26-2015

Re: Disabling Kerberos on Cloudera EXpress 5.5.1: HBase issue

Thanks a lot, I will give it a try and let you know.

Your help is much appreciated.

Explorer
Posts: 12
Registered: ‎06-05-2017

Re: Disabling Kerberos on Cloudera EXpress 5.5.1: HBase issue

Recently i have disabled kerberos on cluster and was able to start hbase master . followed above steps and resolved my issue. Thanks a lot for your suggestions
Announcements