Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

shyamshaw — Thu, 02 Jan 2020 16:54:33 GMT

Hi Team,

I am trying to use FairScheduler in HDP 3.0.1. As per https://stackoverflow.com/questions/38208845/switching-from-capacity-scheduler-to-fair-scheduler-on-hortonworks-data-platform it seems to be possible in HDP 3.0. I have changed the schedule claa and set yarn.scheduler.fair.allocation.file in custom yarn-site in ambari (screenshot attached). But when I restart ResourceManager it is failing with below error.

2020-01-02 16:21:54,134 INFO fair.AllocationFileLoaderService (AllocationFileLoaderService.java:reloadAllocations(230)) - Loading allocation file file:/etc/hadoop/conf/fair-scheduler.xml 2020-01-02 16:21:54,221 INFO monitor.SchedulingMonitor (SchedulingMonitor.java:serviceInit(60)) - Initializing SchedulingMonitor=SchedulingMonitor (ProportionalCapacityPreemptionPolicy) 2020-01-02 16:21:54,221 INFO capacity.ProportionalCapacityPreemptionPolicy (ProportionalCapacityPreemptionPolicy.java:init(160)) - Preemption monitor:org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy 2020-01-02 16:21:54,222 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service SchedulingMonitor (ProportionalCapacityPreemptionPolicy) failed in state INITED org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler not instance of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler at org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.init(ProportionalCapacityPreemptionPolicy.java:165) at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor.serviceInit(SchedulingMonitor.java:61) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.updateSchedulingMonitors(SchedulingMonitorManager.java:93) at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.initialize(SchedulingMonitorManager.java:123) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1422) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1230) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1512)

Any help on this issue.

Re: Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

Shelton — Thu, 02 Jan 2020 19:54:22 GMT

@shyamshaw

I am already answering a similar question see this thread

https://community.cloudera.com/t5/Support-Questions/Unable-to-start-Node-Manager/td-p/285976

Please go through the thread and update me with what isn't working, I will answer to both threads soon

Re: Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

shyamshaw — Fri, 03 Jan 2020 05:30:44 GMT

Hi @Shelton

Thank you for the update. It seems there are some configuration needs to be added/changed in YARN. I will perform the changes and update you in case of any failures.

Re: Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

shyamshaw — Tue, 07 Jan 2020 07:03:25 GMT

@Shelton Sorry for late response. I followed the thread and was able to configure Fair Scheduler successfully.

Thank you for your help.

Cheers.

question Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1 in Support Questions

Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

Re: Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

Re: Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

Re: Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1