Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

Solved Go to solution

Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

Expert Contributor

Hi Team,

I am trying to use FairScheduler in HDP 3.0.1. As per https://stackoverflow.com/questions/38208845/switching-from-capacity-scheduler-to-fair-scheduler-on-... it seems to be possible in HDP 3.0. I have changed the schedule claa and set yarn.scheduler.fair.allocation.file in custom yarn-site in ambari (screenshot attached). But when I restart ResourceManager it is failing with below error.

2020-01-02 16:21:54,134 INFO fair.AllocationFileLoaderService (AllocationFileLoaderService.java:reloadAllocations(230)) - Loading allocation file file:/etc/hadoop/conf/fair-scheduler.xml
2020-01-02 16:21:54,221 INFO monitor.SchedulingMonitor (SchedulingMonitor.java:serviceInit(60)) - Initializing SchedulingMonitor=SchedulingMonitor (ProportionalCapacityPreemptionPolicy)
2020-01-02 16:21:54,221 INFO capacity.ProportionalCapacityPreemptionPolicy (ProportionalCapacityPreemptionPolicy.java:init(160)) - Preemption monitor:org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy
2020-01-02 16:21:54,222 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service SchedulingMonitor (ProportionalCapacityPreemptionPolicy) failed in state INITED
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler not instance of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
at org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.init(ProportionalCapacityPreemptionPolicy.java:165)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor.serviceInit(SchedulingMonitor.java:61)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.updateSchedulingMonitors(SchedulingMonitorManager.java:93)
at org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitorManager.initialize(SchedulingMonitorManager.java:123)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1422)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:817)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1230)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:317)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1512)

 image.png

Any help on this issue.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

Mentor

@shyamshaw 

 

I am already answering a similar question see this thread 

https://community.cloudera.com/t5/Support-Questions/Unable-to-start-Node-Manager/td-p/285976

Please go through the thread and update me with what isn't working, I will answer to both threads soon

 

3 REPLIES 3
Highlighted

Re: Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

Mentor

@shyamshaw 

 

I am already answering a similar question see this thread 

https://community.cloudera.com/t5/Support-Questions/Unable-to-start-Node-Manager/td-p/285976

Please go through the thread and update me with what isn't working, I will answer to both threads soon

 

Re: Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

Expert Contributor

Hi @Shelton 

Thank you for the update. It seems there are some configuration needs to be added/changed in YARN. I will perform the changes and update you in case of any failures.

Re: Switching from Capacity Scheduler to Fair Scheduler on HDP 3.0.1

Expert Contributor

@Shelton Sorry for late response. I followed the thread and was able to configure Fair Scheduler successfully.

Thank you for your help.

Cheers.

Don't have an account?
Coming from Hortonworks? Activate your account here