question Re: Limit ressource allocate to HBase query based on user in Archives of Support Questions (Read Only)

Limit ressource allocate to HBase query based on user

michelsumbul — Fri, 12 Feb 2016 22:05:45 GMT

Hi,

Is there a way to specify how much ressource an user will be able to use for his HBase query?

The objectif is to be able to define group A that can use 40 % of the ressouce and group B 60%, thoses 2 groups have to query the same hbase cluster.

Some thing like we can do with the yarn queue manager but on every query lunch by every user?

Thanks for the info,

Michel

Re: Limit ressource allocate to HBase query based on user

nsabharwal — Fri, 12 Feb 2016 22:10:52 GMT

@Michel Sumbul

Very good question...You may have to read this jira https://issues.apache.org/jira/browse/HBASE-11598

Re: Limit ressource allocate to HBase query based on user

nsabharwal — Fri, 12 Feb 2016 22:11:11 GMT

Tagging experts @Enis @Josh Elser

Re: Limit ressource allocate to HBase query based on user

aervits — Sat, 13 Feb 2016 03:17:04 GMT

@Michel Sumbul the feature is called rquest throttling. It is available in HBase 1.1 and thus in HDP 2.3. More info https://blogs.apache.org/hbase/entry/the_hbase_request_throttling_feature

The throttle can then be set from the HBase shell, like so:

hbase> set_quota TYPE => THROTTLE, USER => 'uname', LIMIT => '100req/sec'

hbase> set_quota TYPE => THROTTLE, TABLE => 'tbl', LIMIT => '10M/sec'

hbase> set_quota TYPE => THROTTLE, NAMESPACE => 'ns', LIMIT => 'NONE'

our in-house engineer responded in a similar post recently https://community.hortonworks.com/questions/1821/hbase-quota-management.html

Re: Limit ressource allocate to HBase query based on user

nsabharwal — Sat, 13 Feb 2016 03:27:23 GMT

@Michel Sumbul

The above jira provides the detail history on throttling.

Also, see if you can research on following parameter if you are using phoenix

phoenix.query.maxTenantMemoryPercentage

https://phoenix.apache.org/tuning.html

Re: Limit ressource allocate to HBase query based on user

michelsumbul — Tue, 16 Feb 2016 17:49:43 GMT

Thanks for the fast reply guys! 🙂

Re: Limit ressource allocate to HBase query based on user

sumit_nigam — Thu, 22 Sep 2016 11:21:53 GMT

@Artem Ervits @Neeraj Sabharwal - I am trying to leverage size-based throttling but keep getting ThrottlingException when I start hbase, even when there is hardly any data in hbase.

I am sure this is some mis-configuration from my end but I cannot seem to find that out. Any inputs would be appreciated. Just to also add there is some correlation here between number of pre-splits and throttling size limit because the error shows up only when number of pre-splits are more.

Details : Hbase version : 1.1.2, Number of region servers :4, Number of regions : 116, HeapMemory for Region Server : 2GB

Quotas set :

TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10G/sec, SCOPE => MACHINE

TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10G/sec, SCOPE => MACHINE

Region server stack trace (notice below that the error is about read size limit exceeded, and later the size of scan is only 28 (bytes?). Stack trace:-

2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] quotas.RegionServerQuotaManager: Throttling exception for user=root table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit exceeded - wait 0.00sec

2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId: 52 service: ClientService methodName: Scan size: 28 connection: 10.65.141.170:42806 org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit exceeded - wait 0.00sec at org.apache.hadoop.hbase.quotas.ThrottlingException.throwThrottlingException(ThrottlingException.java:107)