Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hbase region servers crash with Out of Memory

avatar
Super Collaborator

not sure what parameter i have to change

Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2484)
... 5 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
1 ACCEPTED SOLUTION

avatar
Super Collaborator

it happens when Ambari is upgraded to 2.2

this is a known issue -https://issues.apache.org/jira/browse/AMBARI-13325

Disable bucket cache by erasing the values from these propeerties from ambari - and restart the Hbase service. - hbase.bucketcache.ioengine - hbase.bucketcache.percentage.in.combinedcache - hbase.bucketcache.size

View solution in original post

5 REPLIES 5

avatar
Guru

It seems related to the offheap setting. You should look at the -XXMaxDirectMemorySize parameter in hbase-env.sh and make sure that it is large enough to hold the offheap block cache (if configured) and some offset from HDFS client offheap buffers (usually small).

avatar
Super Collaborator

@Enis

value is set to 4G isn't big enough? we hardly use hbase

avatar
Guru

It depends on the bucket cache size settings for HBase. If you have configured the bucket cache bigger than 4G (which is all off-heap) you will get this once the cache grows.

avatar
Super Collaborator

it happens when Ambari is upgraded to 2.2

this is a known issue -https://issues.apache.org/jira/browse/AMBARI-13325

Disable bucket cache by erasing the values from these propeerties from ambari - and restart the Hbase service. - hbase.bucketcache.ioengine - hbase.bucketcache.percentage.in.combinedcache - hbase.bucketcache.size

avatar
Super Collaborator

We just encountered this upgrading from 2.1.1 to 2.2.2

We removed the settings referenced and that got us around the issue, but are those settings important?