question Re: Why can't I run more than 1 query in parallel in Hive? in Archives of Support Questions (Read Only)

Why can't I run more than 1 query in parallel in Hive?

guillaume_roger — Sun, 18 Aug 2019 00:11:18 GMT

I have a small one node hdp2.6 cluster (8 CPUs, 32GB ram), and I cannot run more than 1 query at a time, although I was pretty sure that I configures the relevant settings to allow more than one container.

The relevant configs are:

yarn-site/yarn.nodemanager.resource.memory-mb = 27660
yarn-site/yarn.scheduler.minimum-allocation-mb = 5532
yarn-site/yarn.scheduler.maximum-allocation-mb = 27660
mapred-site/mapreduce.map.memory.mb = 5532
mapred-site/mapreduce.reduce.memory.mb = 11064
mapred-site/mapreduce.map.java.opts = -Xmx4425m
mapred-site/mapreduce.reduce.java.opts =  -Xmx8851m
mapred-site/yarn.app.mapreduce.am.resource.mb = 11059
mapred-site/yarn.app.mapreduce.am.command-opts = -Xmx8851m -Dhdp.version=${hdp.version}

hive-site/hive.execution.engine = tez
hive-site/hive.tez.container.size = 5532
hive-site/hive.auto.convert.join.noconditionaltask.size = 1546859315
tez-site/tez.runtime.unordered.output.buffer.size-mb = 414
tez-interactive-site/tez.am.resource.memory.mb = 5532
tez-site/tez.am.resource.memory.mb = 5532
tez-site/tez.task.resource.memory.mb = 5532
tez-site/tez.runtime.io.sort.mb = 1351
hive-site/hive.tez.java.opts = -server -Xmx4425m -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseParallelGC -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps

capacity-scheduler/yarn.scheduler.capacity.resource-calculator = org.apache.hadoop.yarn.util.resource.DominantResourceCalculatororg.apache.hadoop.yarn.util.resource.DominantResourceCalculator
yarn-site/yarn.nodemanager.resource.cpu-vcores = 6
yarn-site/yarn.scheduler.maximum-allocation-vcores = 6

mapred-site/mapreduce.map.output.compress = true
hive-site/hive.exec.compress.intermediate = true
hive-site/hive.exec.compress.output = true


hive-interactive-env/enable_hive_interactive = false

Which if I understand it well, gives 5GB per container.

If I run a hive query, it will use 5GB, 1 core, leaving about 15GB and 5 cores for the rest. I do not understand why the next query cannot start at the same time.

Any help would be much welcome.

Re: Why can't I run more than 1 query in parallel in Hive?

vasubramanian — Tue, 04 Jul 2017 13:24:24 GMT

@Guillaume Roger: The issue seems to be happening as the maximum application master limit has been set to 20%.

Maximum percent of resources in the cluster which can be used to run application masters - controls number of concurrent active applications. Limits on each queue are directly proportional to their queue capacities and user limits.

Since the limit here has been reached, I guess the application is not able to spawn another application master and hence the query is not running. Can we try increasing the parameter yarn.scheduler.capacity.maximum-am-resource-percent / yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent to 35% and see if it resolves the issue?

Re: Why can't I run more than 1 query in parallel in Hive?

guillaume_roger — Tue, 04 Jul 2017 20:22:13 GMT

@Vani, Thanks for your answer.

I do not see an immediate change, but I carry on looking in this direction.

What would be a good logical value for this maximum-am-resource-percent?

Currently the AM memory (tez-site/tez.am.resource.memory.mb) is set to the min container size (5GB in my case). Does that make sense?

Re: Why can't I run more than 1 query in parallel in Hive?

vasubramanian — Tue, 04 Jul 2017 21:32:31 GMT

@Guillaume Roger

If AM memory (tez-site/tez.am.resource.memory.mb) is set to the min container size, only 1 application can be launched at any given point of time. Please tune the property depending on your requirement .

Re: Why can't I run more than 1 query in parallel in Hive?

guillaume_roger — Thu, 06 Jul 2017 17:44:32 GMT

@Vani I am trying to understand what will this memory be used for. My understanding is that:

any application will require its own AM
one AM will use 1 container only
tez-site/tez.am.resource.memory.mb defines the memory usable by the total of all AM

So logically

all AM memory should never be more than half of the available memory (for the worst case scenario where all application only use one container)
I should allocate in tez-site/tez.am.resource.memory.mb (minimum container size * expected number of applications)

Could you confirm my understanding?

Re: Why can't I run more than 1 query in parallel in Hive?

vasubramanian — Mon, 10 Jul 2017 11:55:15 GMT

@Guillaume Roger

Hoping the attached link will help:-

https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html

Re: Why can't I run more than 1 query in parallel in Hive?

arun_pnu — Thu, 11 Jul 2019 07:06:05 GMT

@Guillaume Roger, did you manage to sort your issue out. I am facing the same problem i.e unable to run 2 concurrent jobs although there is enought memory to spin new containers.

Any help will be highly appreciated..

Re: Why can't I run more than 1 query in parallel in Hive?

rhj — Wed, 08 Jul 2020 06:37:40 GMT

Stuck here with the same issue on HDP 3.0.1, still searching the correct parameters to set.

Platform is Comunity Edition on PowerPC

Re: Why can't I run more than 1 query in parallel in Hive?

asish — Wed, 08 Jul 2020 08:38:08 GMT

There are 2 things forground thread and background thread.

Foreground thread is used for parsing/complilation.The hive.server2.async.exec.threads determine how many concurrent threads can Hiveserver2 handle,it defaults to 500.

Background thread comes after compliation/parsing phase,where it runs Tez jobs. This depends on number of Tez AM .

In this case, you need to increase the Tez AM's.

hive.server2.tez.sessions.per.default.queue=2

So if you have one Hiveserver2,it will run 2 background threads.

Please refer below document for more information:

https://community.cloudera.com/t5/Community-Articles/Hive-Understanding-concurrent-sessions-queue-allocation/ta-p/247407

Re: Why can't I run more than 1 query in parallel in Hive?

rhj — Thu, 09 Jul 2020 09:50:48 GMT

just i have hive.server2.tez.sessions.per.default.queue=4 ...

Re: Why can't I run more than 1 query in parallel in Hive?

asish — Thu, 09 Jul 2020 11:00:00 GMT

Can you try running 2 queries simultaneously and check?

Re: can you test 2 queries and check

rhj — Thu, 09 Jul 2020 11:10:26 GMT

yes, here is the screenshot:

query 1 (green) is started and query 2 (yellow) waits until all the jobs of query one are done

Re: can you test 2 queries and check

asish — Thu, 09 Jul 2020 11:13:49 GMT

I have checked the screenshot. This is not application concurrency. Reducer phases1 is waiting for all the mappers to get finished. DAG is decided by the optimizer.

Are you using MAPREDUCE or Tez as an execution engine?

Re: can you test 2 queries and check

rhj — Thu, 09 Jul 2020 11:16:28 GMT

I suppose to run TEZ, because I found all the configurations for TEZ ...

But I'm bloody new in this ... So, supposedly I 'm irgnorant too.

Thanks for your fast responses!

Re: can you test 2 queries and check

asish — Thu, 09 Jul 2020 11:21:15 GMT

In Mapreduce the Reducer output would wait after all ten Mapper is finished. We recommend to use Tez.

Re: can you test 2 queries and check

rhj — Thu, 09 Jul 2020 11:32:00 GMT

In my hive-config (Ambari) are plenty of tez-parameters - so I supposed it is TEZ. I did not found a parameter as 'use tez' or 'use mapreduce' ...

hive.convert.join.bucket.mapjoin.tez is False - may for this ?

My queries are running from beeline

Re: can you test 2 queries and check

asish — Thu, 09 Jul 2020 12:02:21 GMT

This is decided by the optimiser.

Until Mappers are finished for that query,Reducers would not be stated.

Re: can you test 2 queries and check

rhj — Thu, 09 Jul 2020 12:14:04 GMT

I'm searching now the correct information for the framework, where I found yarn and NOT yarn-tez in the mapred-site-xml ...

I'm totally new to this architecture, so I have to try - I did not find a docu apllicable to our installation (hdp 3.0.1 on powerpc) with ambari.

But thanks a lot, at least I understand, that we are NOT using TEZ ...

Re: can you test 2 queries and check

asish — Thu, 09 Jul 2020 12:18:23 GMT

Thank you for the update. Tez fixes this kind of issue.

Re: can you test 2 queries and check

rhj — Thu, 09 Jul 2020 12:45:54 GMT

Unfortunately I'm still stick with the activation of tez under hive.

setting the properties:

mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn-tez</value>
</property>

hive-site.xml

<property>
<name>hive.execution.engine</name>
<value>tez</value>
</property>

adding in my beeline query

set hive.execution.engine=tez; ## in the query (now it is faster!)

still is always running only ONE of the two, still saying Starting task [Stage-1:MAPRED] in parallel

😞