Hi , When i type "hbase.regionserver.lease.period" in Ambari configs the following property got invocated "Number of Fetched Rows when Scanning from Disk" It looks we didn't set "hbase.regionserver.lease.period" in our Ambari conifgs. Thanks. ... View more

Thank you so much. We are getting these below exceptions can you please look at it @mqureshi 02-08-17 10:31:37,615 R- WARN ScannerCallable:290 - Ignore, probably already closed org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '12759' does not exist at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221) at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2494) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:744) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:300) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:287) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:153) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:121) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97) at org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:431) at.HBaseVADashBoardDAO.closeQuitely(HBaseVADashBoardDAO.java:539) at.HBaseVADashBoardDAO.getUiJsonKeysForBrand(HBaseVADashBoardDAO.java:715) at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) .TimerAspect.doBasicProfiling(TimerAspect.java:20) at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy17.getUiJsonKeysForBrand(Unknown Source) .CacheServiceImpl.loadBrandLocationTCells(CacheServiceImpl.java:275) .CacheServiceImpl.afterPropertiesSet(CacheServiceImpl.java:110) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1571) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1509) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:296) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:628) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479) ... View more

Thank you guys @mqureshi @Rajendra M @Artem Ervits ... View more

Hi, We ran the HDFS balancer on one of the node in our environment but it fails to run on single node rather it started on entire cluster. 😞 We ran the balancer on single node by adding "-source -f HOSTS.TXT" to the command Thanks. ... View more

Hi , Yes,load balancer was not enabled before, i just checked yesterday in HBase Master UI regions are not evenly spread across all region servers,some of the region servers are there with very less number of regions. The size of hfile.block.cache.size = 0.25(25%). Yesterday we did some cleanup activity,so HBase average load came-down to 2181.07. Please suggest what all needs to be done. Thanks. ... View more

Hi , Most of our tables will have only 1 column family and hbase.regionserver.global.memstore.size is set to 0.25 and flush size is 128MB. And yesterday we ran "balance_switch true" from hbase shell ,does it do any load balancing of regions ? Thanks. ... View more

Hi , Yes,that's right we are having ~6200+ regions per region server, how to change the setting for having less number of regions per region server ? can you please tell me how to enable load balancer for HBase to spread region servers across all regions equally ? HBase region server heap size is 16GB and Memstore flush size is 128MB. Thanks!! ... View more

Hi, Thanks for your reply!! Regions are spread across region servers uniformly and also i can see HBase Master heap size utilization is going to 85-90% (eventhough heap size of HMaster is 8GB) most of the times in Ambari UI ,In our cluster we mostly rely on Hive,HBase . ... View more