Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

default queue accepting one mr job for hive while tez accepting more than one

Solved Go to solution
Highlighted

default queue accepting one mr job for hive while tez accepting more than one

New Contributor

default queue accepting one mr job for hive while tez accepting more than one.

default execution engine is Tez. in this case default queue running 3 different query parallely in 3 different session.

While i set hive.execution.engine=mr in this case i am able to open multiple hive session but default queue run only one query at a time.

My cluster is big and already tried multiple small job parallely for Tez. but setting hive.execution.engine=mr run only one job at a time in default queue.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: default queue accepting one mr job for hive while tez accepting more than one

Expert Contributor

Do you have the YARN Resource Manager screenshot when you run 3 mapred job?

http://<Active_RM_HOST>:8088/cluster/scheduler

From what i read from screen shots,

Maximum AM Resource 20% i.e. 20% of 391GB = 78GB.

value of: yarn.app.mapreduce.am.resource.mb and tez.am.resource.memory.mb will determine how many AMs can fit in to run concurrently.

7 REPLIES 7

Re: default queue accepting one mr job for hive while tez accepting more than one

Expert Contributor

Hi @Alpesh Virani

There are several possibilities. When you have multiple hive sessions open with execution engine "mr", can you tell us:

1. How much resources you have used / available to run in "default" queue ?

Check this on YARN RM UI > Scheduler

2. If you have 100% available in "default" queue, check the "am" container size "yarn.app.mapreduce.am.resource.mb" and check the "Maximum AM Resource" for "default" queue,

see if queue has enough resources to run multiple "am" "mr" containers.

Re: default queue accepting one mr job for hive while tez accepting more than one

New Contributor

screen-shot-2018-03-14-at-83913-am.png

See th attached file it is inheriting all the resource and it is 100%.

Re: default queue accepting one mr job for hive while tez accepting more than one

Expert Contributor

Hi @Alpesh Virani

Can you also share the Resource Manager UI screenshot. This will tell what is the actual usage for your queue.

http://<Active_RM_HOST>:8088/cluster/scheduler

Re: default queue accepting one mr job for hive while tez accepting more than one

New Contributor

screen-shot-2018-03-14-at-84702-am.png

If you see i am able to run 3 TEZ job. but when i run 3 mapred job in hive.execution.engine=mr in hive. it is accepting one at a time despite enough space.

Re: default queue accepting one mr job for hive while tez accepting more than one

Expert Contributor

Do you have the YARN Resource Manager screenshot when you run 3 mapred job?

http://<Active_RM_HOST>:8088/cluster/scheduler

From what i read from screen shots,

Maximum AM Resource 20% i.e. 20% of 391GB = 78GB.

value of: yarn.app.mapreduce.am.resource.mb and tez.am.resource.memory.mb will determine how many AMs can fit in to run concurrently.

Re: default queue accepting one mr job for hive while tez accepting more than one

New Contributor

Thanks you very much for your prompt reply. I am able to run map reduce with 300 GB usage. I think i need to look some other parameter. your hint may help me. currently i can not run it but will do it soon.

Re: default queue accepting one mr job for hive while tez accepting more than one

Expert Contributor

Please accept an Answer, so that we can mark this request close.

Don't have an account?
Coming from Hortonworks? Activate your account here