how to set the yarn application priority in hive ?

Question

Yarn's application priority can be found in yarn 8088 resource manager's website,

How do the priority work in yarn?

Now the version of CDH I use is 6.3.2.

Hadoop 3.0.0

Hive 2.1.1

I use hive on spark.

Can I use the config to manage the application priority?

For some reason some hive sqls should have high application priority when those sqls are appending and run ahead of the other. Instead of running the same time and equal share the compute resources.

When I set the following config in Hive, it seems do not work well in Yarn.

MapReduce	"-Dmapreduce.job.priority=xx"
Flink	"-yD yarn.applicaiton.priority=xx"
Spark	"spark.yarn.priority=xx"

In hive sql."set spark.yarn.priority=10;"

It does not work ...

AsimShaikh · Accepted Answer

Hi @bulbcat

- Spark version used in CDH 6.3.2 is Spark 2.4.0 [1]

- Also spark.yarn.priority will only effect from spark-3.0 [2]

[1]https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_63_packaging.html

[2]https://github.com/apache/spark/pull/27856

If you want to update priority for application during runtime you can use below

yarn application -updatePriority 10 -appId application_xxxx_xx

Thanks!

bulbcat · Answer

Thanks a lot.

This "yarn application -updatePriority 10 -appId application_xxxx_xx"

seems a config of yarn.

It does not work for spark 2.x in CDH 6.3.2 either.

Does it the same reason which means the 'Application Priority' must match the yarn version with spark version?

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.