Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

avatar
Expert Contributor

Hi Team,

I am trying to use FairScheduler in HDP 3.0.1. As per https://stackoverflow.com/questions/38208845/switching-from-capacity-scheduler-to-fair-scheduler-on-... it seems to be possible in HDP 3.0. I have changed the schedule claa and set yarn.scheduler.fair.allocation.file in custom yarn-site in ambari (screenshot attached). But when I restart ResourceManager it is failing with below error.

2020-01-02 16:21:54,134 INFO fair.AllocationFileLoaderService (AllocationFileLoaderService.java:reloadAllocations(230)) - Loading allocation file file:/etc/hadoop/conf/fair-scheduler.xml
2020-01-02 16:21:54,221 INFO monitor.SchedulingMonitor (SchedulingMonitor.java:serviceInit(60)) - Initializing SchedulingMonitor=SchedulingMonitor (ProportionalCapacityPreemptionPolicy)
2020-01-02 16:21:54,221 INFO capacity.ProportionalCapacityPreemptionPolicy (ProportionalCapacityPreemptionPolicy.java:init(160)) - Preemption monitor:org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy
2020-01-02 16:21:54,222 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service SchedulingMonitor (ProportionalCapacityPreemptionPolicy) failed in state INITED
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler not instance of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
at org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.init(ProportionalCapacityPreemptionPolicy.java:165)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor.serviceInit(SchedulingMonitor.java:61)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.updateSchedulingMonitors(SchedulingMonitorManager.java:93)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.initialize(SchedulingMonitorManager.java:123)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1422)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1230)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1512)

 image.png

Any help on this issue.

1 ACCEPTED SOLUTION

avatar
Master Mentor

@shyamshaw 

 

I am already answering a similar question see this thread 

https://community.cloudera.com/t5/Support-Questions/Unable-to-start-Node-Manager/td-p/285976

Please go through the thread and update me with what isn't working, I will answer to both threads soon

 

View solution in original post

3 REPLIES 3

avatar
Master Mentor

@shyamshaw 

 

I am already answering a similar question see this thread 

https://community.cloudera.com/t5/Support-Questions/Unable-to-start-Node-Manager/td-p/285976

Please go through the thread and update me with what isn't working, I will answer to both threads soon

 

avatar
Expert Contributor

Hi @Shelton 

Thank you for the update. It seems there are some configuration needs to be added/changed in YARN. I will perform the changes and update you in case of any failures.

avatar
Expert Contributor

@Shelton Sorry for late response. I followed the thread and was able to configure Fair Scheduler successfully.

Thank you for your help.

Cheers.