Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HBase on Sandbox 2.3.2 fails

avatar
Rising Star

I have a HDP 2.3.2 sandbox up and running on VMWare fusion. I am able to start both HBase master and region server but then they fail after few hours.

It seems like region server loses its connection to zookeeper for some reason. Here is the error that i get

2016-02-07 22:32:46,693 FATAL [main-EventThread] regionserver.HRegionServer: ABORTING region server sandbox.hortonworks.com,16020,1454868273283: regionserver:16020-0x150af1fbaec0164, quorum=sandbox.hortonworks.com:2181, baseZNode=/hbase-unsecure regionserver:16020-0x150af1fbaec0164 received expired from ZooKeeper, aborting org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:606) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:517) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

Please let me know if i need to make any setting changes in sandbox.

1 ACCEPTED SOLUTION

avatar
Super Guru

Lots of good comments here too, but also consider:

1. Reducing the swappiness value (10 is a safe bet) http://askubuntu.com/questions/103915/how-do-i-configure-swappiness

2. Disabling transparent huge pages (THP) https://access.redhat.com/solutions/46111

Both of these OS-level properties can exhibit the same symptoms as you would see when running out of memory in the JVM.

View solution in original post

7 REPLIES 7

avatar
Master Mentor
@Chakra

It can be because of low memory. Are you using HBase ? How much memory allocated to vm?

avatar
Rising Star

i have 8 GB allocated to the VM. You think thats sufficient?

avatar
Master Mentor
@Chakra

are you doing large imports? You may need to tune your JVM and GC pauses. Also look at zookeeper.session.timeout property.

avatar
Rising Star

I am not having any activity on HBase. I wanted to make sure its stable before i build a demo on top of it. Should i add a custom HBase site property name

zookeeper.session.timeout

and set it to 60000?

avatar
Master Mentor

turn off the other services when you run HBase like metrics, hive, oozie, atlas, etc. I've run HBase without issues on Sandbox, perhaps you have a bad image? Try re-importing again. @Chakra

avatar
Master Mentor

Zookeeper expiration can happen due to low memory on the node or GCs that are too long

You can add session timeout in your hbase-site.xml like this

<property>

<name>zookeeper.session.timeout</name>

<value>60000</value>

</property>

avatar
Super Guru

Lots of good comments here too, but also consider:

1. Reducing the swappiness value (10 is a safe bet) http://askubuntu.com/questions/103915/how-do-i-configure-swappiness

2. Disabling transparent huge pages (THP) https://access.redhat.com/solutions/46111

Both of these OS-level properties can exhibit the same symptoms as you would see when running out of memory in the JVM.