Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hbase throttling errors.

Highlighted

Hbase throttling errors.

Contributor

Hi

Our application is unable to scan or read from hbase tables, when throttling is set. The ThrottlingException seems to have some correlation to pre-splits because it shows up as and when pre-splits for tables are increased. This is despite the fact that the table under consideration is mostly empty.

 

We have tried both rate limiters already - average and fixed. Can't understand why read rate limit exceeds when there is hardly any data in hbase. Anyone has faced this issue before? One thing we are planning to try is reattempt the operations when we get this error. Not sure, if that is helpful or not. However, would like to know how are others able to get this working.


Setup Details: 

Hbase version : 1.1.2

Number of region servers :4 

Number of regions : 116 

HeapMemory for Region Server : 2GB 


Quotas set : 


TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10G/sec, SCOPE => MACHINE 


TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10G/sec, SCOPE => MACHINE 


Region server stack trace (notice below that the error is about read size limit exceeded, and later the size of scan is only 28 (bytes?):

 

2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] quotas.RegionServerQuotaManager: Throttling exception for user=root table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit exceeded - wait 0.00sec 

2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId: 52 service: ClientService methodName: Scan size: 28 connection: 10.65.141.170:42806 

org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit exceeded - wait 0.00sec 

at org.apache.hadoop.hbase.quotas.ThrottlingException.throwThrottlingException(ThrottlingException.java:107) 
at org.apache.hadoop.hbase.quotas.ThrottlingException.throwReadSizeExceeded(ThrottlingException.java:101) 
at org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:139) 
at org.apache.hadoop.hbase.quotas.DefaultOperationQuota.checkQuota(DefaultOperationQuota.java:59) 
at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(RegionServerQuotaManager.java:180) 
at org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(RegionServerQuotaManager.java:125) 
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2265) 
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) 
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) 
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) 
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) 
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) 
at java.lang.Thread.run(Thread.java:745)

 

Thanks,
Sumit

1 REPLY 1

Re: Hbase throttling errors.

Contributor

One thing we could debug here is that the table was created via Apache Phoenix. The table is empty. When we query the table via phoenix shell, we get the ThrottlingException. The same works fine through hbase shell.

 

I use Hbase 1.1.2 and Phoenix 4.7.

 

Anyone has faced this issue before? 

Don't have an account?
Coming from Hortonworks? Activate your account here