Skip to main content
shyamshaw
Gold ⭐️
Gold ⭐️
January 2, 2020
Solved

Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

Hi Team,

I am trying to use FairScheduler in HDP 3.0.1. As per https://stackoverflow.com/questions/38208845/switching-from-capacity-scheduler-to-fair-scheduler-on-hortonworks-data-platform it seems to be possible in HDP 3.0. I have changed the schedule claa and set yarn.scheduler.fair.allocation.file in custom yarn-site in ambari (screenshot attached). But when I restart ResourceManager it is failing with below error.

2020-01-02 16:21:54,134 INFO fair.AllocationFileLoaderService (AllocationFileLoaderService.java:reloadAllocations(230)) - Loading allocation file file:/etc/hadoop/conf/fair-scheduler.xml
2020-01-02 16:21:54,221 INFO monitor.SchedulingMonitor (SchedulingMonitor.java:serviceInit(60)) - Initializing SchedulingMonitor=SchedulingMonitor (ProportionalCapacityPreemptionPolicy)
2020-01-02 16:21:54,221 INFO capacity.ProportionalCapacityPreemptionPolicy (ProportionalCapacityPreemptionPolicy.java:init(160)) - Preemption monitor:org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy
2020-01-02 16:21:54,222 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service SchedulingMonitor (ProportionalCapacityPreemptionPolicy) failed in state INITED
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler not instance of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
at org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.init(ProportionalCapacityPreemptionPolicy.java:165)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor.serviceInit(SchedulingMonitor.java:61)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.updateSchedulingMonitors(SchedulingMonitorManager.java:93)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.initialize(SchedulingMonitorManager.java:123)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1422)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1230)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1512)

 image.png

Any help on this issue.

Shelton
SheltonAnswer
Gold ⭐️⭐️⭐️
January 2, 2020

@shyamshaw 

 

I am already answering a similar question see this thread 

https://community.cloudera.com/t5/Support-Questions/Unable-to-start-Node-Manager/td-p/285976

Please go through the thread and update me with what isn't working, I will answer to both threads soon

 

3 replies

Shelton
SheltonAnswer
Gold ⭐️⭐️⭐️
January 2, 2020

@shyamshaw 

 

I am already answering a similar question see this thread 

https://community.cloudera.com/t5/Support-Questions/Unable-to-start-Node-Manager/td-p/285976

Please go through the thread and update me with what isn't working, I will answer to both threads soon

 

Translate
shyamshaw
Gold ⭐️
shyamshawAuthor
Gold ⭐️
January 3, 2020

Hi @Shelton 

Thank you for the update. It seems there are some configuration needs to be added/changed in YARN. I will perform the changes and update you in case of any failures.

Translate