HBASE Master Heap Size Recommendation

[#1]

Hi All ,

Is there any rule of thumb for setting up HBASE master heap size, in our present environment HBASE Master heap size is 8GB and the average load of regions per region server is ~6200+.

Is there any need for changing Master heap size based on regions per region server is very high.

Thanks in advance.

[#2]

Thank you so much.

We are getting these below exceptions can you please look at it @mqureshi

02-08-17 10:31:37,615 R- WARN ScannerCallable:290 - Ignore, probably already closed

org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '12759' does not exist

at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221)

at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206)

at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2494)

at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)

at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)

at java.lang.Thread.run(Thread.java:744)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)

at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)

at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:300)

at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:287)

at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:153)

at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:121)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)

at org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:431)

at.HBaseVADashBoardDAO.closeQuitely(HBaseVADashBoardDAO.java:539)

at.HBaseVADashBoardDAO.getUiJsonKeysForBrand(HBaseVADashBoardDAO.java:715)

at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)

.TimerAspect.doBasicProfiling(TimerAspect.java:20)

at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)

at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

at com.sun.proxy.$Proxy17.getUiJsonKeysForBrand(Unknown Source)

.CacheServiceImpl.loadBrandLocationTCells(CacheServiceImpl.java:275)

.CacheServiceImpl.afterPropertiesSet(CacheServiceImpl.java:110)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)

at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)

at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)

at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)

[#3]

For your lease exception, what is the value of hbase.regionserver.lease.period? When a client connects to HBase, it gets a lease and client needs to report back within this time period. If it doesn't then it is considered dead and you run into this exception. On way to avoid this is to increase the lease period, but that's just addressing a sympton. question is, why is a client taking more than 60 seconds (assuming that's the value you have set - default of 60 seconds).

Check the following link. This is a really good discussion on this issue.

http://mail-archives.apache.org/mod_mbox/hbase-user/201209.mbox/%3CCAOcnVr3R-LqtKhFsk8Bhrm-YW2i9O6J6...

Here is from documentation:

In some situations clients that fetch data from a RegionServer get a LeaseException instead of the usual Section 12.5.1, “ScannerTimeoutException or UnknownScannerException”. Usually the source of the exception is org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230) (line number may vary - in your case this is line 221 but its exact same exception). It tends to happen in the context of a slow/freezing RegionServer#next call. It can be prevented by having hbase.rpc.timeout > hbase.regionserver.lease.period.

[#4]

Hi ,

When i type "hbase.regionserver.lease.period" in Ambari configs the following property got invocated "Number of Fetched Rows when Scanning from Disk"

It looks we didn't set "hbase.regionserver.lease.period" in our Ambari conifgs.

Thanks.