Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hbase region servers crash with Out of Memory

Solved Go to solution

Hbase region servers crash with Out of Memory

Expert Contributor

not sure what parameter i have to change

Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2484)
... 5 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Hbase region servers crash with Out of Memory

Expert Contributor

it happens when Ambari is upgraded to 2.2

this is a known issue -https://issues.apache.org/jira/browse/AMBARI-13325

Disable bucket cache by erasing the values from these propeerties from ambari - and restart the Hbase service. - hbase.bucketcache.ioengine - hbase.bucketcache.percentage.in.combinedcache - hbase.bucketcache.size

5 REPLIES 5

Re: Hbase region servers crash with Out of Memory

Guru

It seems related to the offheap setting. You should look at the -XXMaxDirectMemorySize parameter in hbase-env.sh and make sure that it is large enough to hold the offheap block cache (if configured) and some offset from HDFS client offheap buffers (usually small).

Re: Hbase region servers crash with Out of Memory

Expert Contributor

@Enis

value is set to 4G isn't big enough? we hardly use hbase

Highlighted

Re: Hbase region servers crash with Out of Memory

Guru

It depends on the bucket cache size settings for HBase. If you have configured the bucket cache bigger than 4G (which is all off-heap) you will get this once the cache grows.

Re: Hbase region servers crash with Out of Memory

Expert Contributor

it happens when Ambari is upgraded to 2.2

this is a known issue -https://issues.apache.org/jira/browse/AMBARI-13325

Disable bucket cache by erasing the values from these propeerties from ambari - and restart the Hbase service. - hbase.bucketcache.ioengine - hbase.bucketcache.percentage.in.combinedcache - hbase.bucketcache.size

Re: Hbase region servers crash with Out of Memory

Super Collaborator

We just encountered this upgrading from 2.1.1 to 2.2.2

We removed the settings referenced and that got us around the issue, but are those settings important?

Don't have an account?
Coming from Hortonworks? Activate your account here