HBase on Sandbox 2.3.2 fails

[#1]

I have a HDP 2.3.2 sandbox up and running on VMWare fusion. I am able to start both HBase master and region server but then they fail after few hours.

It seems like region server loses its connection to zookeeper for some reason. Here is the error that i get

2016-02-07 22:32:46,693 FATAL [main-EventThread] regionserver.HRegionServer: ABORTING region server sandbox.hortonworks.com,16020,1454868273283: regionserver:16020-0x150af1fbaec0164, quorum=sandbox.hortonworks.com:2181, baseZNode=/hbase-unsecure regionserver:16020-0x150af1fbaec0164 received expired from ZooKeeper, aborting org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:606) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:517) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

Please let me know if i need to make any setting changes in sandbox.

[#2]

@Chakra

It can be because of low memory. Are you using HBase ? How much memory allocated to vm?

[#3]

i have 8 GB allocated to the VM. You think thats sufficient?

[#4]

@Chakra

are you doing large imports? You may need to tune your JVM and GC pauses. Also look at zookeeper.session.timeout property.

[#5]

I am not having any activity on HBase. I wanted to make sure its stable before i build a demo on top of it. Should i add a custom HBase site property name

zookeeper.session.timeout

and set it to 60000?

[#6]

turn off the other services when you run HBase like metrics, hive, oozie, atlas, etc. I've run HBase without issues on Sandbox, perhaps you have a bad image? Try re-importing again. @Chakra

[#7]

Zookeeper expiration can happen due to low memory on the node or GCs that are too long

You can add session timeout in your hbase-site.xml like this

<property>

<name>zookeeper.session.timeout</name>

<value>60000</value>

</property>

[#8 · Accepted solution]

Lots of good comments here too, but also consider:

1. Reducing the swappiness value (10 is a safe bet) http://askubuntu.com/questions/103915/how-do-i-configure-swappiness

2. Disabling transparent huge pages (THP) https://access.redhat.com/solutions/46111

Both of these OS-level properties can exhibit the same symptoms as you would see when running out of memory in the JVM.