Created 02-12-2016 02:05 PM
Hi,
Is there a way to specify how much ressource an user will be able to use for his HBase query?
The objectif is to be able to define group A that can use 40 % of the ressouce and group B 60%, thoses 2 groups have to query the same hbase cluster.
Some thing like we can do with the yarn queue manager but on every query lunch by every user?
Thanks for the info,
Michel
Created 02-12-2016 07:17 PM
@Michel Sumbul the feature is called rquest throttling. It is available in HBase 1.1 and thus in HDP 2.3. More info https://blogs.apache.org/hbase/entry/the_hbase_request_throttling_feature
The throttle can then be set from the HBase shell, like so:
hbase> set_quota TYPE => THROTTLE, USER => 'uname', LIMIT => '100req/sec'
hbase> set_quota TYPE => THROTTLE, TABLE => 'tbl', LIMIT => '10M/sec'
hbase> set_quota TYPE => THROTTLE, NAMESPACE => 'ns', LIMIT => 'NONE'
our in-house engineer responded in a similar post recently https://community.hortonworks.com/questions/1821/hbase-quota-management.html
Created 02-12-2016 02:10 PM
Very good question...You may have to read this jira https://issues.apache.org/jira/browse/HBASE-11598
Created 02-12-2016 07:27 PM
The above jira provides the detail history on throttling.
Also, see if you can research on following parameter if you are using phoenix
phoenix.query.maxTenantMemoryPercentage
Created 09-22-2016 04:21 AM
@Artem Ervits @Neeraj Sabharwal - I am trying to leverage size-based throttling but keep getting ThrottlingException when I start hbase, even when there is hardly any data in hbase.
I am sure this is some mis-configuration from my end but I cannot seem to find that out. Any inputs would be appreciated. Just to also add there is some correlation here between number of pre-splits and throttling size limit because the error shows up only when number of pre-splits are more.
Details : Hbase version : 1.1.2, Number of region servers :4, Number of regions : 116, HeapMemory for Region Server : 2GB
Quotas set :
TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10G/sec, SCOPE => MACHINE
TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10G/sec, SCOPE => MACHINE
Region server stack trace (notice below that the error is about read size limit exceeded, and later the size of scan is only 28 (bytes?). Stack trace:-
2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] quotas.RegionServerQuotaManager: Throttling exception for user=root table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit exceeded - wait 0.00sec
2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId: 52 service: ClientService methodName: Scan size: 28 connection: 10.65.141.170:42806 org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit exceeded - wait 0.00sec at org.apache.hadoop.hbase.quotas.ThrottlingException.throwThrottlingException(ThrottlingException.java:107)
Created 02-12-2016 02:11 PM
Tagging experts @Enis @Josh Elser
Created 02-12-2016 07:17 PM
@Michel Sumbul the feature is called rquest throttling. It is available in HBase 1.1 and thus in HDP 2.3. More info https://blogs.apache.org/hbase/entry/the_hbase_request_throttling_feature
The throttle can then be set from the HBase shell, like so:
hbase> set_quota TYPE => THROTTLE, USER => 'uname', LIMIT => '100req/sec'
hbase> set_quota TYPE => THROTTLE, TABLE => 'tbl', LIMIT => '10M/sec'
hbase> set_quota TYPE => THROTTLE, NAMESPACE => 'ns', LIMIT => 'NONE'
our in-house engineer responded in a similar post recently https://community.hortonworks.com/questions/1821/hbase-quota-management.html
Created 02-16-2016 09:49 AM
Thanks for the fast reply guys! 🙂