Member since
01-24-2021
5
Posts
0
Kudos Received
0
Solutions
11-28-2022
11:09 PM
Thanks a lot. This " yarn application -updatePriority 10 -appId application_xxxx_xx " seems a config of yarn. It does not work for spark 2.x in CDH 6.3.2 either. Does it the same reason which means the 'Application Priority' must match the yarn version with spark version?
... View more
11-24-2022
06:11 PM
Yarn's application priority can be found in yarn 8088 resource manager's website, How do the priority work in yarn? Now the version of CDH I use is 6.3.2. Hadoop 3.0.0 Hive 2.1.1 I use hive on spark. Can I use the config to manage the application priority? For some reason some hive sqls should have high application priority when those sqls are appending and run ahead of the other. Instead of running the same time and equal share the compute resources. When I set the following config in Hive, it seems do not work well in Yarn. MapReduce "-Dmapreduce.job.priority=xx" Flink "-yD yarn.applicaiton.priority=xx" Spark "spark.yarn.priority=xx" In hive sql."set spark.yarn.priority=10;" It does not work ...
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Spark
-
Apache YARN
01-24-2021
09:59 PM
hi, seems the same error . Excuse me , Where is the log of cm yarn usage aggretion logs ? I also set the pool rules and hive some error with user like admin which has no group matching. User admin can not submit job to yarn ,but the other normal user can do it .
... View more
01-24-2021
07:54 PM
hi , In the develop enviroment , there is a service named CM yarn usage aggregation runs in yarn per hour . It can be found in jobhistoryserver's web ui . but in the test env , there is no more . The difference between this two is that develop start with root , test env start with a user with previlege sudo . how to find the starting log of CMyarnusageaggregation to debug the problems ? The log-aggretion properties are both set to enabled. Much Thanks!
... View more
Labels:
- Labels:
-
Apache YARN