Limit ressource allocate to HBase query based on user

[#1]

Hi,

Is there a way to specify how much ressource an user will be able to use for his HBase query?

The objectif is to be able to define group A that can use 40 % of the ressouce and group B 60%, thoses 2 groups have to query the same hbase cluster.

Some thing like we can do with the yarn queue manager but on every query lunch by every user?

Thanks for the info,

Michel

[#2]

@Michel Sumbul

Very good question...You may have to read this jira https://issues.apache.org/jira/browse/HBASE-11598

[#3]

@Michel Sumbul

The above jira provides the detail history on throttling.

Also, see if you can research on following parameter if you are using phoenix

phoenix.query.maxTenantMemoryPercentage

https://phoenix.apache.org/tuning.html

[#4]

@Artem Ervits @Neeraj Sabharwal - I am trying to leverage size-based throttling but keep getting ThrottlingException when I start hbase, even when there is hardly any data in hbase.

I am sure this is some mis-configuration from my end but I cannot seem to find that out. Any inputs would be appreciated. Just to also add there is some correlation here between number of pre-splits and throttling size limit because the error shows up only when number of pre-splits are more.

Details : Hbase version : 1.1.2, Number of region servers :4, Number of regions : 116, HeapMemory for Region Server : 2GB

Quotas set :

TABLE => ns1:table1 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10G/sec, SCOPE => MACHINE

TABLE => ns2:table2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT => 10G/sec, SCOPE => MACHINE

Region server stack trace (notice below that the error is about read size limit exceeded, and later the size of scan is only 28 (bytes?). Stack trace:-

2016-09-17 22:35:40,674 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] quotas.RegionServerQuotaManager: Throttling exception for user=root table=ns1:table1 numWrites=0 numReads=0 numScans=1: read size limit exceeded - wait 0.00sec

2016-09-17 22:35:40,676 DEBUG [B.defaultRpcServer.handler=55,queue=1,port=58526] ipc.RpcServer: B.defaultRpcServer.handler=55,queue=1,port=58526: callId: 52 service: ClientService methodName: Scan size: 28 connection: 10.65.141.170:42806 org.apache.hadoop.hbase.quotas.ThrottlingException: read size limit exceeded - wait 0.00sec at org.apache.hadoop.hbase.quotas.ThrottlingException.throwThrottlingException(ThrottlingException.java:107)

[#5]

Tagging experts @Enis @Josh Elser

[#6 · Accepted solution]

@Michel Sumbul the feature is called rquest throttling. It is available in HBase 1.1 and thus in HDP 2.3. More info https://blogs.apache.org/hbase/entry/the_hbase_request_throttling_feature

The throttle can then be set from the HBase shell, like so:

hbase> set_quota TYPE => THROTTLE, USER => 'uname', LIMIT => '100req/sec'

hbase> set_quota TYPE => THROTTLE, TABLE => 'tbl', LIMIT => '10M/sec'

hbase> set_quota TYPE => THROTTLE, NAMESPACE => 'ns', LIMIT => 'NONE'

our in-house engineer responded in a similar post recently https://community.hortonworks.com/questions/1821/hbase-quota-management.html

[#7]

Thanks for the fast reply guys! 🙂