Support Questions

Find answers, ask questions, and share your expertise

HBASE Master failed to restart (related to zookeeper?)

avatar
Explorer

I've just completed with CM and CDH installation and trying to add HBase service to the Master node. This is not production environment no data has been loaded yet; pretty much a clean install.

 

After adding Hbase service the first to Master Node, I got a bad health alert saying that my disk space is not sufficient. So, what I tried to do was to move hbase log directory to a different location using symlink. I realized that when doing this I forgot to stop the Hbase service. After I'm done with the symlink, I restart the Hbase service and fail. I got the error below.

 

Question, the Hbase service configuration at CM don't have option to move or update the log directory. What is your suggestion to do this in the future? thanks in advance.

 

Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2775)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:184)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2789)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:489)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:468)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1233)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1211)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:472)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2770)
... 5 more
1 ACCEPTED SOLUTION

avatar
The error is originating from ZooKeeper. Which directory did you move:
/var/log or /var/log/hbase? Did you try restarting the entire cluster?

Regards,
Gautam Gopalakrishnan

View solution in original post

3 REPLIES 3

avatar
The error is originating from ZooKeeper. Which directory did you move:
/var/log or /var/log/hbase? Did you try restarting the entire cluster?

Regards,
Gautam Gopalakrishnan

avatar
Explorer

/var/log/hbase

 

I've restarted the entire cluster. Now it's up again.

 

But the problem still remains that I'm getting bad health alert because of /var/log disk space is insufficient. I want to change the log directory for hbase to my /apps/ext/var/log directory. But how can I do that?

 

There are no log storage setting for Hbase in CM.

 

Thanks for the reply.

avatar
Explorer

Nevermind, the log storage setting for HBase is available on CM. I didn't realilze those arrows next to category label can be expanded. And there will be 'Log' category which I can update the value.