question Re: Hbase region servers crash with Out of Memory in Archives of Support Questions (Read Only)

Hbase region servers crash with Out of Memory

chrsvarma — Thu, 14 Jan 2016 08:15:51 GMT

not sure what parameter i have to change

Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2484)
... 5 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)

Re: Hbase region servers crash with Out of Memory

Enis — Thu, 14 Jan 2016 08:19:11 GMT

It seems related to the offheap setting. You should look at the -XXMaxDirectMemorySize parameter in hbase-env.sh and make sure that it is large enough to hold the offheap block cache (if configured) and some offset from HDFS client offheap buffers (usually small).

Re: Hbase region servers crash with Out of Memory

chrsvarma — Thu, 14 Jan 2016 08:20:55 GMT

@Enis

value is set to 4G isn't big enough? we hardly use hbase

Re: Hbase region servers crash with Out of Memory

chrsvarma — Thu, 14 Jan 2016 09:17:12 GMT

it happens when Ambari is upgraded to 2.2

this is a known issue -https://issues.apache.org/jira/browse/AMBARI-13325

Disable bucket cache by erasing the values from these propeerties from ambari - and restart the Hbase service. - hbase.bucketcache.ioengine - hbase.bucketcache.percentage.in.combinedcache - hbase.bucketcache.size

Re: Hbase region servers crash with Out of Memory

Enis — Fri, 15 Jan 2016 03:14:24 GMT

It depends on the bucket cache size settings for HBase. If you have configured the bucket cache bigger than 4G (which is all off-heap) you will get this once the cache grows.

Re: Hbase region servers crash with Out of Memory

zack_riesland — Wed, 18 May 2016 21:54:13 GMT

We just encountered this upgrading from 2.1.1 to 2.2.2

We removed the settings referenced and that got us around the issue, but are those settings important?