Limit Hive query ressource consumption in Yarn

Question

Hi, I face the following problem.I have a service user say "service_dwh" used for our datawarehouse that queries heavily our data reservoir using hive queries.I have had some cases where due to the query and/or to missing statistics a single hive query could take 100% of the resources available for the "service_dwh" user.I couldn't find a way using capacity scheduler, queues and user limit factor to prevent a single application from taking all the resources for a very long time.Traditional DBMS have mechanism that throttle job resources based of the job duration.That way a long (big) job can't monopolyze ressources for new and potentially shorter jobs for too long.

tj2007 · Answer

Hi @gael__urbauer

Please refer to the below articles and see if this is what you are looking for:

[1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/yarn-allocate-resources/topics/yarn-configure-queue-properties.html [2] https://blog.cloudera.com/yarn-capacity-scheduler/#:~:text=User%20Limit%20Factor%20is%20a,minimum%20capacity%20of%20the%20queue.

Let me know if this helps.

Cheers!

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.