Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBASE Master Heap Size Recommendation

Solved Go to solution
Highlighted

Re: HBASE Master Heap Size Recommendation

Explorer

Thank you so much.

We are getting these below exceptions can you please look at it @mqureshi

02-08-17 10:31:37,615 R- WARN ScannerCallable:290 - Ignore, probably already closed

org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '12759' does not exist

at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221)

at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206)

at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2494)

at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)

at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)

at java.lang.Thread.run(Thread.java:744)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)

at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)

at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:300)

at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:287)

at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:153)

at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:121)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)

at org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:431)

at.HBaseVADashBoardDAO.closeQuitely(HBaseVADashBoardDAO.java:539)

at.HBaseVADashBoardDAO.getUiJsonKeysForBrand(HBaseVADashBoardDAO.java:715)

at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)

.TimerAspect.doBasicProfiling(TimerAspect.java:20)

at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)

at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

at com.sun.proxy.$Proxy17.getUiJsonKeysForBrand(Unknown Source)

.CacheServiceImpl.loadBrandLocationTCells(CacheServiceImpl.java:275)

.CacheServiceImpl.afterPropertiesSet(CacheServiceImpl.java:110)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)

at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628)

at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)

at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)

Re: HBASE Master Heap Size Recommendation

Super Guru

For your lease exception, what is the value of hbase.regionserver.lease.period? When a client connects to HBase, it gets a lease and client needs to report back within this time period. If it doesn't then it is considered dead and you run into this exception. On way to avoid this is to increase the lease period, but that's just addressing a sympton. question is, why is a client taking more than 60 seconds (assuming that's the value you have set - default of 60 seconds).

Check the following link. This is a really good discussion on this issue.

http://mail-archives.apache.org/mod_mbox/hbase-user/201209.mbox/%3CCAOcnVr3R-LqtKhFsk8Bhrm-YW2i9O6J6...

Here is from documentation:

In some situations clients that fetch data from a RegionServer get a LeaseException instead of the usual Section 12.5.1, “ScannerTimeoutException or UnknownScannerException”. Usually the source of the exception is org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230) (line number may vary - in your case this is line 221 but its exact same exception). It tends to happen in the context of a slow/freezing RegionServer#next call. It can be prevented by having hbase.rpc.timeout > hbase.regionserver.lease.period.

Highlighted

Re: HBASE Master Heap Size Recommendation

Explorer

Hi ,

When i type "hbase.regionserver.lease.period" in Ambari configs the following property got invocated "Number of Fetched Rows when Scanning from Disk"

12274-lease.png

It looks we didn't set "hbase.regionserver.lease.period" in our Ambari conifgs.

Thanks.

Don't have an account?
Coming from Hortonworks? Activate your account here