Hbase region servers crash with Out of Memory

[#1]

not sure what parameter i have to change

Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2484)
... 5 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)

[#2]

It seems related to the offheap setting. You should look at the -XXMaxDirectMemorySize parameter in hbase-env.sh and make sure that it is large enough to hold the offheap block cache (if configured) and some offset from HDFS client offheap buffers (usually small).

[#3]

@Enis

value is set to 4G isn't big enough? we hardly use hbase

[#4]

It depends on the bucket cache size settings for HBase. If you have configured the bucket cache bigger than 4G (which is all off-heap) you will get this once the cache grows.

[#5 · Accepted solution]

it happens when Ambari is upgraded to 2.2

this is a known issue -https://issues.apache.org/jira/browse/AMBARI-13325

Disable bucket cache by erasing the values from these propeerties from ambari - and restart the Hbase service. - hbase.bucketcache.ioengine - hbase.bucketcache.percentage.in.combinedcache - hbase.bucketcache.size

[#6]

We just encountered this upgrading from 2.1.1 to 2.2.2

We removed the settings referenced and that got us around the issue, but are those settings important?