Hi - we keep running into this everytime we try refreshing the dynamic resource pool in CDH 5.5.1:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt: Reservation Exceeds Allowed number of nodes: app_id=application_1452220058915_0013 existingReservations=5 totalAvailableNodes=94 reservableNodesRatio=0.05 numAllowedReservations=5
How to get past this issue and have the resource allocated appropriately?
We have the exact same problem with YARN on CDH 5.5.1. At first the problem appeared straight away and we couldn't launch any job (there were all hanging in the UNASSIGNED state) and after YARN restart a few mapred jobs went fine and then it happened again. This error message you mentioned was flooding resource manager's logs when the problem was occurring - it generated 1GB of logs in 5 minutes.
I have switched from Fair to Capacity scheduler for now and I'm looking for a solution to make Fair sched. work again...
Turning off multiple assign fixed the issue for me, its releated to this:
This should probably be reported under known issues for CDH 5.5.1.
Hope it helps!
We are aware of that upstream bug and found it during our internal performance testing, around the time we released CDH 5.5.1
It will be included in an upcoming CDH 5.5 release.