I successfully configured the fair scheduler on the below HDP version
Original scheduler
YARN UI
Default capacity scheduler after deployment of HDP
Pre-emption enabled before the change to fair-scheduler
Grabbed the template fair-scheduler.xml fair-scheduler here I then changed a few values for testing purposes but ensured the is valid using the XML using XML Validator I then copied the fair-scheduler.xml to the $HADOOP_CONF directory and changed the user & permission
# cd /usr/hdp/3.1.0.0-78/hadoop/conf
# chown hdfs:hadoop fair-scheduler.xml
# chmod 644 fair-scheduler.xml
Changed the Scheduler class in the yarn-site.xml see the attached screenshot.
From : yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
To: yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
Added these new values in Custom yarn-site using the relative path default to /usr/hdp/3.1.0.0-78/hadoop/conf
yarn.scheduler.fair.allocation.file=fair-scheduler.xml
Custom yarn-site.xml
Changed the below mandatory parameter to enable the ReservationSystem in the ResourceManager is not enabled by default
yarn.resourcemanager.reservation-system.enable=true
Disable pre-emption
Set the below properties as shown
The yarn-site.xml file contains parameters that determine scheduler-wide options. These properties include the below properties if they don't exist add them in the custom yarn-site
Note
The below property was available so I didn't add
yarn.scheduler.capacity.ordering-policy.priority-utilization.underutilized-preemption.enabled=false
Properties to verify
For my testing I didn't add the below properties, you will notice that above that despite disabling the pre-emption in the Ambari UI the fair schedule shows it's enabled [True] and my queues ain't showing I need to check my fair-scheduler.xml attached is the template I used
yarn.scheduler.fair.assignmultiple=false
yarn.scheduler.fair.sizebasedweight=false
yarn.scheduler.fair.user-as-default-queue=true
yarn.scheduler.fair.preemption=false Note: Do not use preemption when FairScheduler DominantResourceFairness is in
use and node labels are present.
All in all, this shows the fair-scheduler configuration is doable and my RM is up and running !!
I also noticed that the above fair-scheduler was template overwritten when I checked the YARN Queue Manager so that can now allow me to configure a new valid fair-scheduler
Happy Hadooping
Hi @Shelton,
Thanks for the reply. I have been able to change the scheduling mode to Fair-Scheduler, which is great.
However, my application is not running, due to resource allocation issue. I am getting the below standard error.
| [Mon Dec 30 15:02:19 +0530 2019] Application is added to the scheduler and is not yet activated. (Resource request: <memory:1024, vCores:1> exceeds maximum AM resource allowed). |
I am attaching all the relevant screenshots as well as information of my Yarn cluster for your reference.
Please guide me in fixing this issue. Why this issue usually occurs.
My YARN cluster has 2 nodes, scheduling mode as "Fair-Scheduler", minimum allocation of 1 GB/1 vcores and maximum allocation of 15GB/3 vcores and overall memory is 30GB.
Given below is "fair-scheduler.xml" contents:
Below is the custom yarn-site parameters that have been set and the preemption is disabled as well.
Hi @Shelton ,
I have made some progress on this issue. I have modified the fair-scheduler.xml and have set both "maxAMShare" and "queueMaxAMShareDefault" to 0.8 and weight to default value (1.0).
The result: One spark job is running fine. However, I am getting the same error as before on the exceeding of maximum AM resources limit, when I try to run the next job.
The modified fair-scheduler.xml is given below. Please provide your inputs on how to fix this particular issue.
Also, one interesting observation is that, even though the YARN Scheduling mode is showing as "Fair", the Spark Scheduling mode is still showing as "FIFO". Can I set it to "Fair" as well through the program? Since I am setting spark.master as "YARN", I believe the Fair scheduling mode will take precedence over the Spark scheduling mode. Please correct me if I am wrong.
