question Re: HBase on Sandbox 2.3.2 fails in Archives of Support Questions (Read Only)

HBase on Sandbox 2.3.2 fails

csankaraiah — Mon, 08 Feb 2016 19:44:47 GMT

I have a HDP 2.3.2 sandbox up and running on VMWare fusion. I am able to start both HBase master and region server but then they fail after few hours.

It seems like region server loses its connection to zookeeper for some reason. Here is the error that i get

2016-02-07 22:32:46,693 FATAL [main-EventThread] regionserver.HRegionServer: ABORTING region server sandbox.hortonworks.com,16020,1454868273283: regionserver:16020-0x150af1fbaec0164, quorum=sandbox.hortonworks.com:2181, baseZNode=/hbase-unsecure regionserver:16020-0x150af1fbaec0164 received expired from ZooKeeper, aborting org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:606) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:517) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

Please let me know if i need to make any setting changes in sandbox.

Re: HBase on Sandbox 2.3.2 fails

nsabharwal — Mon, 08 Feb 2016 19:49:01 GMT

@Chakra

It can be because of low memory. Are you using HBase ? How much memory allocated to vm?

Re: HBase on Sandbox 2.3.2 fails

aervits — Mon, 08 Feb 2016 20:55:21 GMT

@Chakra

are you doing large imports? You may need to tune your JVM and GC pauses. Also look at zookeeper.session.timeout property.

Re: HBase on Sandbox 2.3.2 fails

Shelton — Mon, 08 Feb 2016 21:25:11 GMT

Zookeeper expiration can happen due to low memory on the node or GCs that are too long

You can add session timeout in your hbase-site.xml like this

<name>zookeeper.session.timeout</name>

</property>

Re: HBase on Sandbox 2.3.2 fails

elserj — Mon, 08 Feb 2016 23:27:32 GMT

Lots of good comments here too, but also consider:

1. Reducing the swappiness value (10 is a safe bet) http://askubuntu.com/questions/103915/how-do-i-configure-swappiness

2. Disabling transparent huge pages (THP) https://access.redhat.com/solutions/46111

Both of these OS-level properties can exhibit the same symptoms as you would see when running out of memory in the JVM.

Re: HBase on Sandbox 2.3.2 fails

csankaraiah — Tue, 09 Feb 2016 00:16:56 GMT

i have 8 GB allocated to the VM. You think thats sufficient?

Re: HBase on Sandbox 2.3.2 fails

csankaraiah — Tue, 09 Feb 2016 00:20:13 GMT

I am not having any activity on HBase. I wanted to make sure its stable before i build a demo on top of it. Should i add a custom HBase site property name

zookeeper.session.timeout

and set it to 60000?

Re: HBase on Sandbox 2.3.2 fails

aervits — Tue, 09 Feb 2016 00:29:57 GMT

turn off the other services when you run HBase like metrics, hive, oozie, atlas, etc. I've run HBase without issues on Sandbox, perhaps you have a bad image? Try re-importing again. @Chakra