Reply
New Contributor
Posts: 8
Registered: ‎08-26-2016

How to set yarn.resourcemanager preemption.max_wait_before_kill ?

[ Edited ]

Hello,

Is it possible to set this parameter from Cloudera Express 5.14.1 UI?

yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill

 

I highly suspect Yarn to kill some spark executor before they can properly end.

 

regards

Julien

Posts: 1,760
Kudos: 379
Solutions: 282
Registered: ‎07-31-2013

Re: How to set yarn.resourcemanager preemption.max_wait_before_kill ?

The property you refer to is for the Capacity Scheduler's preemption settings. Are you using the Capacity Scheduler in your cluster?

For parameters not available in a service's UI, you can typically add it to the safety-valve of the relevant config file. Your quoted configuration is for the ResourceManager, so add it to the following field:

YARN - Configuration - 'ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml' as property name and value in the UI.

Note: CDH and CM primarily supports the Fair Scheduler which has fair-share based preemption controls that can be fine-tuned extensively: http://blog.cloudera.com/blog/2018/06/yarn-fairscheduler-preemption-deep-dive/
Announcements