Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why can't I run more than 1 query in parallel in Hive?

Solved Go to solution
Highlighted

Re: Why can't I run more than 1 query in parallel in Hive?

Contributor

Can you try running 2 queries simultaneously and check?

 

 

 

Highlighted

Re: can you test 2 queries and check

Explorer

000809_20200709_130929_WN00066156@CN2911IT002481.pngyes, here is the screenshot:

query 1 (green) is started and query 2 (yellow) waits until all the jobs of query one are done

 

000808_20200709_130749_WN00066156@CN2911IT002481.png

Highlighted

Re: can you test 2 queries and check

Contributor

I have checked the screenshot. This is not application concurrency. Reducer phases1 is waiting for all the mappers to get finished.  DAG is  decided by the optimizer.

Are you using MAPREDUCE or Tez as an execution engine?

Re: can you test 2 queries and check

Explorer

I suppose to run TEZ, because I found all the configurations for TEZ ...

 

But I'm bloody new in this ... So, supposedly I 'm irgnorant too.

 

Thanks for your fast responses!

 

Highlighted

Re: can you test 2 queries and check

Contributor

In Mapreduce the Reducer output would wait after all ten Mapper is finished. We recommend to use Tez.

View solution in original post

Highlighted

Re: can you test 2 queries and check

Explorer

In my hive-config (Ambari) are plenty of tez-parameters - so I supposed it is TEZ. I did not found a parameter as 'use tez' or 'use mapreduce' ... 

 

hive.convert.join.bucket.mapjoin.tez is False - may for this ?

 

My queries are running from beeline

 

 

Highlighted

Re: can you test 2 queries and check

Contributor

This is decided by the optimiser.

Until Mappers are finished for that query,Reducers would not be stated.

Highlighted

Re: can you test 2 queries and check

Explorer

I'm searching now the correct information for the framework, where I found yarn and NOT yarn-tez in the mapred-site-xml ...

 

I'm totally new to this architecture, so I have to try  - I did not find a docu apllicable to our installation (hdp 3.0.1 on powerpc) with ambari.

 

But thanks a lot, at least I understand, that we are NOT using TEZ ...

 

 

 

Highlighted

Re: can you test 2 queries and check

Contributor

Thank you for the update. Tez fixes this kind of issue.

Highlighted

Re: can you test 2 queries and check

Explorer

Unfortunately I'm still stick with the activation of tez under hive. 

 

setting the properties:

mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn-tez</value>
</property>

 

hive-site.xml

<property>
<name>hive.execution.engine</name>
<value>tez</value>
</property>

 

adding in my beeline query

set hive.execution.engine=tez;  ## in the query (now it is faster!)

 

 

still is always running only ONE of the two, still saying Starting task [Stage-1:MAPRED] in parallel

 

 

 

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here