Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase on Sandbox 2.3.2 fails

Solved Go to solution

HBase on Sandbox 2.3.2 fails

Explorer

I have a HDP 2.3.2 sandbox up and running on VMWare fusion. I am able to start both HBase master and region server but then they fail after few hours.

It seems like region server loses its connection to zookeeper for some reason. Here is the error that i get

2016-02-07 22:32:46,693 FATAL [main-EventThread] regionserver.HRegionServer: ABORTING region server sandbox.hortonworks.com,16020,1454868273283: regionserver:16020-0x150af1fbaec0164, quorum=sandbox.hortonworks.com:2181, baseZNode=/hbase-unsecure regionserver:16020-0x150af1fbaec0164 received expired from ZooKeeper, aborting org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:606) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:517) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

Please let me know if i need to make any setting changes in sandbox.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: HBase on Sandbox 2.3.2 fails

Lots of good comments here too, but also consider:

1. Reducing the swappiness value (10 is a safe bet) http://askubuntu.com/questions/103915/how-do-i-configure-swappiness

2. Disabling transparent huge pages (THP) https://access.redhat.com/solutions/46111

Both of these OS-level properties can exhibit the same symptoms as you would see when running out of memory in the JVM.

7 REPLIES 7
Highlighted

Re: HBase on Sandbox 2.3.2 fails

@Chakra

It can be because of low memory. Are you using HBase ? How much memory allocated to vm?

Re: HBase on Sandbox 2.3.2 fails

Explorer

i have 8 GB allocated to the VM. You think thats sufficient?

Re: HBase on Sandbox 2.3.2 fails

Mentor
@Chakra

are you doing large imports? You may need to tune your JVM and GC pauses. Also look at zookeeper.session.timeout property.

Re: HBase on Sandbox 2.3.2 fails

Explorer

I am not having any activity on HBase. I wanted to make sure its stable before i build a demo on top of it. Should i add a custom HBase site property name

zookeeper.session.timeout

and set it to 60000?

Re: HBase on Sandbox 2.3.2 fails

Mentor

turn off the other services when you run HBase like metrics, hive, oozie, atlas, etc. I've run HBase without issues on Sandbox, perhaps you have a bad image? Try re-importing again. @Chakra

Re: HBase on Sandbox 2.3.2 fails

Mentor

Zookeeper expiration can happen due to low memory on the node or GCs that are too long

You can add session timeout in your hbase-site.xml like this

<property>

<name>zookeeper.session.timeout</name>

<value>60000</value>

</property>

Re: HBase on Sandbox 2.3.2 fails

Lots of good comments here too, but also consider:

1. Reducing the swappiness value (10 is a safe bet) http://askubuntu.com/questions/103915/how-do-i-configure-swappiness

2. Disabling transparent huge pages (THP) https://access.redhat.com/solutions/46111

Both of these OS-level properties can exhibit the same symptoms as you would see when running out of memory in the JVM.

Don't have an account?
Coming from Hortonworks? Activate your account here