Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn doesn't release resources when Hive query (TEZ) has already finished running

Yarn doesn't release resources when Hive query (TEZ) has already finished running

New Contributor

I'm trying to test Hive queries running on the cluster. When I execute 10 simple queries (using Ambari Hive Views / TEZ), I can see them all running in YARN.

However in Ambari Hive View, they finish in 1 minute or so, but it takes a further 4-6 minutes based on the YARN Finish Time for it to be marked with a Final Status of 'Succeeded' in YARN before those resources are allocated to the next queries in line.

After doing some research, I found that it might come down to:

  • tez.session.am.dag.submit.timeout.secs (Currently set as 300s)
  • tez.am.container.idle.release-timeout-max.millis (Currently set as 20,000ms)
  • tez.am.container.idle.release-timeout-min.millis (Currently set as 10,000ms)

It looks like the tez.session.am.dag.submit.timeout.secs is the one that might be impacting me as it sounds like there is a 300s delay (~5 minutes) which is consistent to what I'm saying. However this seems to be the default that is suggested by Hortonworks and other users?

Is there any other setting I can look at tweaking?

Could the tez.session.am.dag.submit.timeout.secs be changed to something like 30 seconds to allow the next query to run - or are there implications to this?

Thanks

2 REPLIES 2

Re: Yarn doesn't release resources when Hive query (TEZ) has already finished running

New Contributor

Answering an old question, not sure if it's already resolved :)

tez.am.container.reuse.enabled @Lucy Trengel , can you tell if it's set to true ? As this will help you in specifying whether container should be reused !


Re: Yarn doesn't release resources when Hive query (TEZ) has already finished running

New Contributor

for my case, tez.am.container.reuse.enabled = True. I ran a simple select * statement from the readonly user, but still the Tez session is in running state for the user.

application_1560323362594_1913 HIVE-9c2f473e-a685-4233-a522-394933d2905d TEZ readonly default RUNNING UNDEFINED 0% http://mydomain:37441/ui/

Can any one tell me what was the issue. How to use Tez to use the resource correctly. Please dont share any document, I am tired. Please post me practical solution.

Thanks...